Phenotype analysis of cellular image data using a deep metric network

ABSTRACT

The present disclosure relates to phenotype analysis of cellular image data using a deep metric network. One example embodiment includes a method. The method includes receiving a target image of a target biological cell having a target phenotype. The method also includes obtaining a semantic embedding associated with the target image. The semantic embedding is generated using a machine-learned, deep metric network model. Further, the method includes obtaining, for each of a plurality of candidate images of candidate biological cells each having a respective candidate phenotype, a semantic embedding associated with the respective candidate image. In addition, the method includes identifying, for each of the semantic embeddings, common morphological variations and reducing, for each of the semantic embeddings based on the identified common morphological variations, effects of nuisances. Even further, the method includes determining, by the computing device, a similarity score for each candidate image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation application claiming priorityto U.S. application Ser. No. 16/133,542, filed Sep. 17, 2018; whichitself is a Continuation-in-Part application claiming priority to U.S.application Ser. No. 15/808,699, filed Nov. 9, 2017; which itself is aContinuation-in-Part application claiming priority to U.S. applicationSer. No. 15/433,027, filed Feb. 15, 2017; the contents of each of whichare hereby incorporated by reference in their entirety.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Understanding mechanisms by which a disease acts can be important whenprescribing a treatment regimen for a patient having such a disease. Forsome diseases, the current state of knowledge may not be at a level thatallows for such a treatment regimen to be developed. Thus, methods ofimproving the level of understanding of disease mechanisms, or ofscreening for effective treatments even while remaining relativelyunknowledgeable about a given disease mechanism, could be useful intreating patients.

Further, machine learning is a field in computing that involves acomputing device training a model using “training data.” There are twoprimary classifications of methods of training models: supervisedlearning and unsupervised learning. In supervised learning, the trainingdata is classified into data types, and the model is trained to look forvariations/similarities among known classifications. In unsupervisedlearning, the model is trained using training data that is unclassified.Thus, in unsupervised learning, the model is trained to identifysimilarities based on unlabeled training data.

Once the model has been trained on the training data, the model can thenbe used to analyze new data (sometimes called “test data”). Based on themodel's training, a computing device can use the trained model toevaluate the similarity of the test data.

There are numerous types of machine-learned models, each having its ownset of advantages and disadvantages. One popular machine-learned modelis an artificial neural network. The artificial neural network involveslayers of structure, each trained to identify certain features of aninput (e.g., an input image, an input sound file, or an input textfile). Each layer may be built upon sub-layers that are trained toidentify sub-features of a given feature. For example, an artificialneural network may identify composite objects within an image based onsub-features such as edges or textures.

Given the current state of computing power, in some artificial neuralnetworks many such sub-layers can be established during training of amodel. Artificial neural networks that include multiple sub-layers aresometimes referred to as “deep neural networks.” In some deep neuralnetworks, there may be hidden layers and/or hidden sub-layers thatidentify composites or superpositions of inputs. Such composites orsuperpositions may not be human-interpretable.

SUMMARY

The specification and drawings disclose embodiments that relate tophenotype analysis of cellular image data using a deep metric network.

In a first aspect, the disclosure describes a method. The methodincludes receiving, by a computing device, a target image of a targetbiological cell having a target phenotype. The method also includesobtaining, by the computing device, a semantic embedding associated withthe target image. The semantic embedding associated with the targetimage is generated using a machine-learned, deep metric network model.Further, the method includes obtaining, by the computing device for eachof a plurality of candidate images of candidate biological cells eachhaving a respective candidate phenotype, a semantic embedding associatedwith the respective candidate image. The semantic embedding associatedwith the respective candidate image is generated using themachine-learned, deep metric network model. In addition, the methodincludes identifying, by the computing device for each of the semanticembeddings, common morphological variations. Still further, the methodincludes reducing, by the computing for each of the semantic embeddingsbased on the identified common morphological variations, effects ofnuisances. Additionally, the method includes determining, by thecomputing device, a similarity score for each candidate image. Thesimilarity score for each candidate image represents a degree ofsimilarity between the target phenotype and the respective candidatephenotype.

In a second aspect, the disclosure describes a non-transitory,computer-readable medium having instructions stored thereon. Theinstructions, when executed by a processor, cause the processor toexecute a method. The method includes receiving, by the processor, atarget image of a target biological cell having a target phenotype. Themethod also includes obtaining, by the processor, a semantic embeddingassociated with the target image. The semantic embedding associated withthe target image is generated using a machine-learned, deep metricnetwork model. Further, the method includes obtaining, by the processorfor each of a plurality of candidate images of candidate biologicalcells each having a respective candidate phenotype, a semantic embeddingassociated with the respective candidate image. The semantic embeddingassociated with the respective candidate image is generated using themachine-learned, deep metric network model. In addition, the methodincludes identifying, by the processor for each of the semanticembeddings, common morphological variations. Still further, the methodincludes reducing, by the processor for each of the semantic embeddingsbased on the identified common morphological variations, effects ofnuisance. Additionally, the method includes determining, by theprocessor, a similarity score for each candidate image. The similarityscore for each candidate image represents a degree of similarity betweenthe target phenotype and the respective candidate phenotype.

In a third aspect, the disclosure describes a method. The methodincludes preparing a multi-well sample plate with a target biologicalcell having a target phenotype and candidate biological cells. Themethod also includes applying a variety of candidate treatment regimensto each of the candidate biological cells. Further, the method includesrecording a target image of the target biological cell. Additionally,the method includes recording candidate images of each of the candidatebiological cells, each having a respective candidate phenotype arisingin response to the candidate treatment regimen being applied. Stillfurther, the method includes receiving, by a computing device, thetarget image and the candidate images. In addition, the method includesobtaining, by the computing device, a semantic embedding associated withthe target image. The semantic embedding associated with the targetimage is generated using a machine-learned, deep metric network model.Even further, the method includes obtaining, by the computing device foreach candidate image, a semantic embedding associated with therespective candidate image. The semantic embedding associated with therespective candidate image is generated using the machine-learned, deepmetric network model. Yet further, the method includes identifying, bythe computing device for each of the semantic embeddings, commonmorphological variations. Still yet further, the method includesreducing, by the computing device for each of the semantic embeddingsbased on the identified common morphological variations, effects ofnuisances. Even yet further, the method includes determining, by thecomputing device, a similarity score for each candidate image. Thesimilarity score for each candidate image represents a degree ofsimilarity between the target phenotype and the respective candidatephenotype. Still yet even further, the method includes selecting, by thecomputing device, a preferred treatment regimen among the variety ofcandidate treatment regimens based on the similarity scores.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is an illustration of a three-image set, according to exampleembodiments.

FIG. 2A is a graphical illustration of an un-normalized set ofphenotypic data, according to example embodiments.

FIG. 2B is a graphical illustration of a normalized set of phenotypicdata, according to example embodiments.

FIG. 2C is a graphical illustration of an un-normalized set ofphenotypic data, according to example embodiments.

FIG. 2D is a graphical illustration of a normalized set of phenotypicdata, according to example embodiments.

FIG. 3A is a graphical illustration of a data set having a candidatephenotype and a target phenotype, according to example embodiments.

FIG. 3B is a graphical illustration of a target phenotype and athreshold similarity score, according to example embodiments.

FIG. 3C is a graphical illustration of patient stratification, accordingto example embodiments.

FIG. 3D is a graphical illustration of patient stratification, accordingto example embodiments.

FIG. 3E is a graphical illustration of patient stratification, accordingto example embodiments.

FIG. 3F is a graphical illustration of patient stratification, accordingto example embodiments.

FIG. 3G is a graphical illustration of patient stratification, accordingto example embodiments.

FIG. 3H is a graphical illustration of a topology enforced duringtraining of a machine-learned, deep metric network model, according toexample embodiments.

FIG. 3I is a graphical illustration of a topology enforced duringtraining of a machine-learned, deep metric network model, according toexample embodiments.

FIG. 3J is a graphical illustration of a topology enforced duringtraining of a machine-learned, deep metric network model, according toexample embodiments.

FIG. 3K is a graphical illustration of a topology enforced duringtraining of a machine-learned, deep metric network model, according toexample embodiments.

FIG. 3L is a graphical illustration of a topology enforced duringtraining of a machine-learned, deep metric network model, according toexample embodiments.

FIG. 4A is a tabular illustration of an un-normalized data set having atarget phenotype and six candidate phenotypes, according to exampleembodiments.

FIG. 4B is a graphical illustration of an un-normalized data set havinga target phenotype and six candidate phenotypes, according to exampleembodiments.

FIG. 4C is a tabular illustration of vector distances between candidatephenotypes and a target phenotype, according to example embodiments.

FIG. 4D is a tabular illustration of a normalized data set having atarget phenotype and six candidate phenotypes, according to exampleembodiments.

FIG. 4E is a graphical illustration of a normalized data set having atarget phenotype and six candidate phenotypes, according to exampleembodiments.

FIG. 4F is a tabular illustration of vector distances between candidatephenotypes and a target phenotype, according to example embodiments.

FIG. 5A is an illustration of an unscaled image of multiple cells,according to example embodiments.

FIG. 5B is an illustration of a scaled image of multiple cells,according to example embodiments.

FIG. 5C is an illustration of a single-cell selection process in animage, according to example embodiments.

FIG. 5D is an illustration of a single-cell image, according to exampleembodiments.

FIG. 5E is an illustration of a patch selection, according to exampleembodiments.

FIG. 6A is an illustration of a composite scientific image, according toexample embodiments.

FIG. 6B is an illustration of a channel that is part of a scientificimage, according to example embodiments.

FIG. 6C is an illustration of a channel that is part of a scientificimage, according to example embodiments.

FIG. 6D is an illustration of a channel that is part of a scientificimage, according to example embodiments.

FIG. 7A is an illustration of a process, according to exampleembodiments.

FIG. 7B is an illustration of a process, according to exampleembodiments.

FIG. 7C is an illustration of a process, according to exampleembodiments.

FIG. 7D is an illustration of a process, according to exampleembodiments.

FIG. 8 is an illustration of anatomical regions of a patient from whichcandidate biological cells and a target biological cell may be acquired,according to example embodiments.

FIG. 9 is an illustration of a multi-well sample plate, according toexample embodiments.

FIG. 10 is an illustration of a method, according to exampleembodiments.

FIG. 11 is an illustration of a method, according to exampleembodiments.

FIG. 12 is an illustration of a system, according to exampleembodiments.

FIG. 13 is an illustration of a method, according to exampleembodiments.

FIG. 14 is an illustration of a method, according to exampleembodiments.

FIG. 15 is an illustration of a method, according to exampleembodiments.

FIG. 16 is an illustration of a method, according to exampleembodiments.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any example embodimentor feature described herein is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexample embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

Furthermore, the particular arrangements shown in the figures should notbe viewed as limiting. It should be understood that other embodimentsmight include more or less of each element shown in a given figure. Inaddition, some of the illustrated elements may be combined or omitted.Similarly, an example embodiment may include elements that are notillustrated in the figures.

I. OVERVIEW

Example embodiments may relate to analysis of perturbed subjects usingsemantic embeddings (e.g., semantic embeddings generated using amachine-learned, deep metric network model) generated based on visualrepresentations of the perturbed subjects. The semantic embeddings mayfurther be used to determine one or more qualities of the perturbedsubjects and/or similarities/differences among the perturbed subjects.

In various embodiments, the subjects may be perturbed using one or moreof a variety of techniques. The way in which the subjects are perturbedmay be the same across all subjects or may vary across multiplesubjects. Further, the perturbation applied to the subjects may alsodepend on the types of subjects being perturbed. For example, biologicalmaterial (e.g., cells) may be perturbed chemically while semiconductormaterials may be perturbed optically or thermally. Other types ofperturbation are also possible and are contemplated herein.

Upon applying perturbations to the subjects, visual representations ofthe subjects may be produced. Visual representations may includecaptured images, recorded videos, or visual representations of otherwisenon-visual data (e.g., a three-dimensional intensity plot where thex-position and y-position correspond to the i^(th) and j^(th) elementsof a covariance matrix, respectively, and the intensity at a given x-yposition corresponds to the value at the (i, j) position of thecovariance matrix).

The visual representations may be produced using at least one imagingmodality. For example, the visual representation may be captured by asensor, such as a camera or a magnetic resonance imaging (MRI) sensor.Alternatively, the visual representation may be produced by a computingdevice based on a set of received data. In still other embodiments, avisual representation may not be produced, but data capable of visualrepresentation may be used directly, without first producing a visualrepresentation. For example, a covariance matrix may be analyzed withoutgenerating a three-dimensional intensity plot. Other types of data thatare amenable to visual representation (e.g., data where a spatiallocation has some correlation to information about the underlyingsubject) besides covariance matrices may also be used.

Using the visual representations, semantic embeddings can be obtainedfor each of the respective visual representations. Such semanticembeddings may be obtained using a machine-learned, deep metric networkmodel, in some embodiments. Then, using the semantic embeddings, each ofthe visual representations can be classified (e.g., into one or moregroups). Based on the classification(s) of each of the visualrepresentations, one or more features of the respective perturbedsubjects may be determined. For example, the type of perturbation(s)applied, the degree of perturbation(s) applied, the effect of theperturbation(s) applied, and/or the type of subject may be determinedbased on the classifications identified for the visual representations.

In some embodiments, the subjects may be biological cells (othersubjects are also possible). Visual representations of the biologicalcells may be produced and semantic embeddings can be obtained in orderto compare the biological cells (e.g., to establish similarities betweenthe biological cells). Establishing similarity between multiplephenotypes may allow for the study of biological pathways. For example,multiple candidate biological cells may be treated with variouscompounds. The time-evolution of the phenotypes of those candidatebiological cells can then be compared to one another. This may allow fora study of mechanisms of action within the candidate biological cells.

In alternate embodiments, target cells having target phenotypes may becompared with candidate cells having candidate phenotypes to establishwhether the candidate cells can be classified as healthy or unhealthycells. For example, if the target cell is known to have a healthy (orunhealthy) phenotype, and it is determined that the candidate cells havea sufficiently similar phenotype to the target phenotype, the candidatecells may be deemed to have a healthy (or unhealthy) phenotype. Further,techniques as described herein may be used to compare cells acquiredfrom various anatomical regions of a patient's body with one another(e.g., to determine if a disease has progressed from one anatomicalregion of a patient's body to another).

Even further, if candidate cells are known to initially have anunhealthy phenotype, the candidate cells may then be treated withvarious candidate treatment regimens (e.g., various candidate treatmentcompounds, various candidate concentrations of a candidate treatmentcompound, or various candidate treatment durations). After treatment,the similarity between the candidate cells and a target cell having ahealthy phenotype may then be determined. The candidate treatmentregimens may then be ranked in successfulness based on the correspondingcandidate cells phenotypic similarity to the target cell. Such atechnique can be used to develop treatment regimens for patients, forexample.

Embodiments may use a machine-learned, deep metric network model (e.g.,executed by a computing device) to facilitate image comparisons ofbiological cells having various cellular phenotypes. Themachine-learned, deep metric network model may be trained, for example,using consumer photographic training data. The consumer photographictraining data may include a number of three-image sets (e.g., 33 millionthree-image sets, 100 million three-image sets, 1 billion three-imagesets, 10 billion three-image sets, or 33 billion three-image sets). Thethree-image sets may be generated based on query results (e.g., userinternet search results). Additionally, the three-image sets may includeimages that depict a wide variety of scenes, not solely biological cellsor even scientific data. For example, one three-image set may includethree images of automobiles, a second three-image set may include threeimages of animals, and a third three-image set may include three imagesof cities.

Further, each three-image set may include a query image, a positiveimage, and a negative image. The query image may be an image that wassearched by a user, for example, and the positive image may have beenidentified by the user as being more similar to the query image than thenegative image was to the query image. Based on these three-image sets,a computing device may refine the machine-learned, deep metric networkmodel. Refining the model may include developing one or more semanticembeddings that describe similarities between images. For example, asemantic embedding may have multiple dimensions (e.g., 64 dimensions)that correspond to various qualities of an image (e.g., shapes,textures, image content, relative sizes of objects, and perspective).The dimensions of the semantic embeddings could be eitherhuman-interpretable or non-human interpretable. In some embodiments, forexample, one or more of the dimensions may be superpositions ofhuman-interpretable features.

While the machine-learned, deep metric network may have been trainedusing consumer photographic data, the model can be applied to data whichis not consumer photographic (e.g., images of cells or other scientificimages). The use of a machine-learned, deep metric network model ontypes of data other than those on which it was trained is sometimesreferred to as “transfer learning.” In order to use the machine-learned,deep metric network model to compare scientific images, the scientificimages may be converted or transformed to a format that is commensuratewith the model. Converting the scientific images (e.g., target images oftarget biological cells or candidate images of candidate biologicalcells) may include scaling or cropping the respective scientific image(e.g., such that the respective scientific image has a size and/or anaspect ratio that can be compared using the model) or convertingchannels of the scientific images to grayscale. Additionalpre-processing may occur prior to using the machine-learned, deep metricnetwork model for phenotype comparison (e.g., the scientific image maybe cropped around a nucleus, such that only one cell is within thescientific image).

Two scientific images may then be compared (e.g., by a computing device)by comparing semantic embeddings generated for the images using themachine-learned, deep metric network model. For example, one scientificimage may be a candidate image of a candidate biological cell having acandidate phenotype and a second scientific image may be a target imageof a target biological cell having a target phenotype. The two imagesmay be compared, using their semantic embeddings, to determine asimilarity score between the two images. The similarity score mayrepresent how similar the cellular phenotypes depicted in the two imagesare.

To compare the two images, a semantic embedding may be obtained for eachimage using the machine-learned, deep metric network model (e.g., by acomputing device). The semantic embeddings may have dimensions thatcorrespond to dimensions associated with the machine-learned, deepmetric network model developed during training. Obtaining a semanticembedding for each image may include, in some embodiments, obtaining asemantic embedding for each channel within the image and thenconcatenating the single-channel semantic embeddings into a unifiedsemantic embedding for the entire image.

In embodiments where multiple images are recorded and analyzed using themachine-learned, deep metric network model, normalization (e.g., typicalvariation normalization) can be performed. Normalization may includescaling and/or shifting the values of one or more of the dimensions ofthe semantic embeddings in some or all of the images (e.g., the valuesmay be scaled and/or shifted in a given dimension based on negativecontrol groups). Further, the values may be scaled and/or shifted suchthat the distribution of values across all images for certain dimensionsmay have specified characteristics. For example, the values may bescaled and/or shifted such that the distribution for a given dimensionhas zero-mean and unit variance. In some embodiments, the normalizationmay be performed after using principal component analysis (PCA).Additionally, in some embodiments, all dimensions may be scaled and/orshifted to have zero-mean and unit variance (i.e., the dimensions may be“whitened”).

After obtaining semantic embeddings for the images using themachine-learned, deep metric network model, similarity scores can becalculated. The similarity score for a candidate image/phenotype maycorrespond to the vector distance in n-dimensional space (e.g., where nis the number of dimensions defined within the semantic embeddings ofthe machine-learned, deep metric network model) between the candidateimage/phenotype and the target image/phenotype. In alternateembodiments, the similarity score may correspond to an inverse of thevector distance in n-dimensional space. In still other embodiments,similarity scores may also be calculated between two candidate images oreven between two channels within the same image.

After calculating one or more similarity scores, the similarity scoresmay be analyzed. For example, each of the similarity scores may becompared against a threshold similarity score, with similarity scoresgreater than (or less than) or equal to the threshold similarity scorecorresponding to candidate images of candidate cells having candidatephenotypes that are deemed to be the same as the target phenotype of thetarget biological cell in the target image. In other embodiments, thecandidate images may be ranked by similarity score. In such embodiments,the highest similarity score may correspond to a candidate biologicalcell with a candidate phenotype that is most similar to the targetphenotype. If the candidate biological cells were treated with variouscandidate treatment regimens, and the target phenotype represents ahealthy phenotype, then the candidate treatment regimen used to producethe candidate phenotype corresponding to the highest similarity scoremay be identified as a potentially effective treatment regimen thatcould be applied to a patient.

Additionally, using semantic embeddings determined for target images oftarget biological cells and candidate images of candidate biologicalcells, biological cells may be grouped into a plurality of phenotypicstrata. For example, candidate biological images that have values forone or more dimensions (e.g., within a respective semantic embedding)that are of a threshold similarity (e.g., based on the vector distancebetween the corresponding candidate phenotypes of the candidatebiological images) with one another may be grouped into the samephenotypic stratum. Such a phenotypic stratum may be determinedaccording to a predetermined threshold distance. Alternatively, such aphenotypic stratum may be based on a density of candidate phenotypeswithin a certain region of a corresponding multi-dimensional space(e.g., such a density indicating a similarity among phenotypes of thecandidate biological cells in the corresponding candidate images).

Further, in some embodiments, some of the candidate biological cells inthe candidate images may have had candidate treatment compounds appliedor induced genetic mutations. By comparing the semantic embeddings ofsuch candidate biological cells, one or more candidate treatmentcompounds can be identified as mimicking one or more geneticmodifications (e.g., mutations). For example, if the same phenotypicstratum results after applying a candidate treatment compound as resultsafter applying a given genetic mutation, the candidate treatmentcompound may effectively “mimic” the genetic modification. Similarly, ifapplying a candidate treatment compound results in a candidatebiological cell moving from one phenotypic stratum (e.g., representing afirst stage of a disease) to another phenotypic stratum (e.g.,representing a second stage of a disease), and applying a geneticmutation results in the same movement between phenotypic strata, thecandidate treatment compound may “mimic” the genetic modification.

II. EXAMPLE PROCESSES

The following description and accompanying drawings will elucidatefeatures of various example embodiments. The embodiments provided are byway of example, and are not intended to be limiting. As such, thedimensions of the drawings are not necessarily to scale. Further,various subjects are described throughout as being perturbed and/orimaged. For example, one or more biological cells may be described asexample subjects. It is understood that, throughout, whenever anoperation is described as being performed on an example subject, (suchas a biological cell) or an image or other visual representation of suchan example subject, such an operation could be equally performed onother types of subjects (e.g., fibroblasts, malaria cells, yeast cells,yeast cultures, bacteria, bacterial cultures, fungus, fungal cultures,cancer cells, blood cells, malarial parasites, mitochondria, nuclei,axons, dendrites, induced pluripotent stem cells, biological cells froma given region of an organism, biological cells from a given tissue ofan organism, biological cells from a given organ of an organism,biological cells from a given system of an organism, biological cellensembles, tissues, organs, organoids, biological systems, organisms,groups of organisms, ecosystems, chemical compounds, crystals, metallicglasses, mixtures of metallic salts, semiconductors, metals,dielectrics, graphene, microelectromechanical systems (MEMS), ornanoelectromechanical systems (NEMS)).

FIG. 1 is an illustration of a three-image set 100, according to exampleembodiments. The three-image set 100 includes a query image 102, apositive image 104, and a negative image 106. The three-image set 100may be an example of a three-image set used as training date for amachine-learned, deep metric network model (e.g., used duringunsupervised training of the machine-learned, deep metric network). Eachimage in the three-image set 100 may be a three channel (e.g., with ared channel, a green channel, and a blue channel) consumer photograph.Further, each channel may have an 8-bit depth (e.g., may have a decimalvalue between 0 and 255, inclusive, for each pixel in each channel thatindicates the intensity of the pixel). Other numbers of channels and bitdepths are also possible, in various embodiments. In some embodiments,each image in a three-image set, or even each image across allthree-image sets, may have a standardized size (e.g., 200 pixels by 200pixels).

Further, the query image 102, the positive image 104, and the negativeimage 106 may be internet search results. In some embodiments, based onuser feedback, the positive image 104 may be more similar to the queryimage 102 than the positive image 104 is to the negative image 106. Insome embodiments, the three-image sets used as training data for themachine-learned, deep metric network may not depict biologicalcells/phenotypes. For example, the query image could be a car, thepositive image could be a truck, and the negative image could be anairplane.

Various features of the query image 102, the positive image 104, and thenegative image 106 may influence the user feedback. Some examplefeatures include shapes, textures, image content, relative sizes ofobjects, and perspective depicted in the query image 102, the positiveimage 104, and the negative image 106. Other features are also possible.In the example illustrated in FIG. 1, the positive image 104 includesthe same set of shapes as the query image 102, but in a differentarrangement. Conversely, the negative image 106 includes a different setof shapes from the query image 102.

In some embodiments, a computing device may use multiple three-imagesets (e.g., about 100 million total images or about 100 millionthree-image sets) to define semantic embeddings within themachine-learned, deep metric network model. For example, the computingdevice may establish 64 dimensions within semantic embeddings of themachine-learned, deep metric network model for each channel of theimages in the three-image sets. In other embodiments, other numbers ofdimensions are also possible. The dimensions may contain informationcorresponding to the various features of the three-image sets used totrain the machine-learned, deep metric network model. Further, forimages assigned semantic embeddings according to the machine-learned,deep metric network model, the semantic embeddings can be used toanalyze the degree of similarity between two images (e.g., based on avector distance in a multi-dimensional space defined by the semanticembeddings between the two images, i.e., a similarity score).

Additionally, the features within the three-image sets used by acomputing device to update the machine-learned, deep metric model may beselected based on the positive image 104, the query image 102, and thenegative image 106 (as opposed to pre-identified by a programmer, forinstance). In other words, the process used by a computing device totrain the machine-learned, deep metric network model may includeunsupervised learning. The features used to define various dimensions ofthe semantic embeddings may be human-interpretable (e.g., colors, sizes,textures, or shapes) or non-human-interpretable (e.g., superpositions ofhuman-interpretable features), in various embodiments.

FIG. 2A is a graphical illustration of an un-normalized set ofphenotypic data, according to example embodiments. For example, thephenotypic data may correspond to biological cells in candidate images,target images, and/or control group images being analyzed using amachine-learned, deep metric network model. The phenotypic data mayrepresent the number (“counts”) of biological cells from the group ofbiological cells (e.g., candidate biological cells) that have a givenvalue for a given dimension (X₁) of a semantic embedding. Asillustrated, the number of biological cells having a given value for thegiven dimension (X₁) of the semantic embedding may be pseudo-continuous(or approximated as a continuous function). In other words, any value ofthe dimension (X₁) within a given range is occupied by at least one ofthe biological cells in the group. However, in alternate embodiments,only discrete values of the dimension (X₁) may be possible, in whichcase the illustration of the distribution of the data may more closelyresemble a bar graph.

The distribution illustrated in FIG. 2A may have a mean value (<X₁>) anda standard deviation (σ_(X) ₁ ). As illustrated, the mean value (<X₁>)is positive. The distribution illustrated in FIG. 2A may represent adistribution that is not normalized. For example, the distributionillustrated in FIG. 2A may be a distribution as it is recorded fromimages of candidate biological cells after obtaining a semanticembedding, but prior to any other processing (e.g., normalization).

FIG. 2B is a graphical illustration of a normalized set of phenotypicdata, according to example embodiments. The data set illustrated in FIG.2B may be a modified equivalent of the data set illustrated in FIG. 2A(e.g., after normalization). Such a modification may be a result ofscaling (e.g., by multiplication) and/or shifting (e.g., by addition) ofthe values of semantic embeddings of each image in the data set by agiven amount (e.g., a scaling factor). In some embodiments, thephenotypic data may be normalized (e.g., scaled or shifted) based onphenotypic data from a negative control group (e.g., an unperturbedcontrol group) or a positive control group (e.g., a control grouptreated with a candidate compound that yields a known response or acontrol group having phenotypes of a known disease state). For example,it may be known that a control group's phenotypic data has an expectedmean and an expected standard deviation. If, after obtaining thesemantic embeddings, the control group's phenotypic data does not havethe expected mean and/or the expected standard deviation, the controlgroups phenotypic data may be normalized (e.g., scaled and/or shifted byone or more factors). Thereafter, phenotypic data for sets of cellsother than the control group (e.g., from a candidate group of cells) maybe normalized in a way corresponding to the normalization of the controlgroup (e.g., scaled and/or shifted by the same one or more factors).

As illustrated, the data set in FIG. 2B may be normalized such that themean value (<X′₁>) of the dimension (X₁) of the semantic embedding iszero (i.e., the distribution of data is zero-centered). Other meanvalues are also possible in alternate embodiments (e.g., a positive meanvalue, a negative mean value, a unity mean value, etc.). Further, thedata set illustrated in FIG. 2B may be normalized such that the standarddeviation (σ′_(X) ₁ ) of the dimension (X₁) of the semantic embedding is1.0. This may correspond to the data set also having a unit variance(i.e., σ′² _(X) ₁ =1.0) for the dimension (X₁) of the semanticembedding.

A similar normalization to that illustrated in FIG. 2B may be applied tomultiple dimensions of the semantic embedding. By normalizing values ineach dimension to a standardized mean and variance, each dimension mayhave equal weighting (i.e., equal influence) on a distance between twophenotypes as each other dimension. Said another way, each dimension mayequally impact a resulting similarity score. Further, the normalizationillustrated in FIG. 2B may prevent aberrant or extraneous phenotypicresults from overly influencing the impact score (e.g., a largevariation with respect to a target phenotype in a single dimension of acandidate phenotype may not outweigh a series of relatively smallvariations with respect to the target phenotype in alternate dimensionsof the candidate phenotype).

When comparing the normalized set of phenotypic data illustrated in FIG.2B to the non-normalized set of phenotypic data illustrated in FIG. 2A,it is apparent that the normalized mean (<X′₁>) has a value of zero (asopposed to a positive value), and the standard deviation (σ′_(X) ₁ ) wasreduced (e.g., from 1.5 to 1.0). An analogous normalization isillustrated graphically in FIGS. 2C and 2D for a second dimension (X₂)of the semantic embedding. In FIG. 2C, an un-normalized set of thephenotypic data has a negative mean value (<X₂>) and a greater than unitstandard deviation (σ_(X) ₂ ) for the second dimension (X₂). Oncenormalized, as illustrated in FIG. 2D, the set of phenotypic data mayhave a mean of zero (<X′₂>) and a unit variance (σ′² _(X) ₂ ) of 1.0. Insome embodiments, normalizing the phenotypic data illustrated in FIG. 2Dmay involve scaling the data by a value (i.e., a scaling factor) lessthan 1.0 and shifting the data by a positive value (i.e., a positiveshifting value).

In some embodiments, a normalization process may include only adding ashift to the phenotypic data (e.g., to adjust the mean of the data).Alternatively, the normalization process may include only scaling thephenotypic data (e.g., to adjust the standard deviation of the data). Invarious embodiments, various dimensions of the phenotypic data for agiven semantic embedding may be shifted and/or scaled differently fromone another. Further, in some embodiments, one or more of the dimensionsof the phenotypic data for a given semantic embedding may not benormalized at all. The data may not be normalized if the phenotypic datafor one of the dimensions inherently has the desired statisticaldistribution to match with the other dimensions. Alternatively, the datamay not be normalized so that the values of the phenotypic data for agiven dimension either intentionally over-influence or intentionallyunder-influence similarity scores with respect to the rest of thedimensions of the semantic embedding.

As illustrated in FIGS. 2A-2D, both the un-normalized and the normalizeddistributions of the phenotypic data for a given dimension (e.g., X₁ orX₂) may be approximately Gaussian. In some embodiments, however, theun-normalized and the normalized distributions may be pseudo-Gaussian ornon-Gaussian. In some embodiments, the phenotypic data for somedimensions may be Gaussian while the phenotypic data for otherdimensions may be non-Gaussian. Further, in some embodiments, theun-normalized distributions for a given dimension may be non-Gaussianand may be normalized in such a way that the normalized distributionsfor the given dimension may be approximately or actually Gaussian.Various embodiments may have various graphical shapes for the phenotypicdata in various dimensions.

In addition to the normalization illustrated in FIGS. 2A-2D, atransformation of the phenotypic data may be performed to ensure thatthe dimensions generated for a given semantic embedding are orthogonalto one another (i.e., to prevent certain dimensions from being linearcombinations of other dimensions). This may prevent redundant data frominfluencing similarity scores, for example. In alternate embodiments,orthogonal transformations of the dimensions may occur during thetraining of the machine-learned, deep metric network (i.e., rather thanduring phenotypic data analysis).

The orthogonal transformations may include performing PCA, for example.PCA may include calculating eigenvectors and/or eigenvalues of acovariance matrix defined by the phenotypic data in each dimension. Insome embodiments, the orthogonal transformation may also include adimensionality reduction. Having fewer dimensions within a semanticembedding may conserve memory within a storage device of a computingdevice (e.g., within a volatile memory or a non-volatile memory) bypreventing as much data from being stored to describe a phenotypic dataset. Again, the above steps may be performed during the training of themachine-learned, deep metric network. Additionally or alternatively,normalizing the data may include performing a whitening transform on thephenotypic data (e.g., to transform the phenotypic data such that it hasan identity covariance matrix).

FIG. 3A is a graphical illustration of a data set 300 having a candidatephenotype and a target phenotype, according to example embodiments. Thecandidate phenotype may correspond to a candidate image of a candidatebiological cell. Similarly, the target phenotype may correspond to atarget image of a target biological cell. For both the candidatephenotype and the target phenotype, the values of each of the dimensionsmay be found by analyzing the candidate image and the target image,respectively, using the machine-learned, deep metric network model.Further, the values of each of the dimensions illustrated in FIG. 3A maybe normalized or un-normalized. The candidate phenotype and the targetphenotype are depicted as vectors, each having a respective firstdimension (X₁) value and a respective second dimension (X₂) value.

As illustrated in FIG. 3A, the semantic embedding used to define thecandidate phenotype and the target phenotype has two dimensions (namely,X₁ and X₂). FIG. 3A is provided to illustrate an example embodiment. Invarious other embodiments, other numbers of dimensions may also bepossible. For example, in some embodiments, the semantic embeddings mayinclude 16, 32, 64, 128, 192, or 256, or 320 dimensions.

Also illustrated in FIG. 3A is a vector representing the similarityscore between the target phenotype vector and the candidate phenotypevector. The similarity score vector may be equal to the target phenotypevector minus the candidate phenotype vector. The magnitude (i.e.,length) of the similarity score vector may correspond to a value of thesimilarity score. Said another way, the distance between the targetphenotype vector and the candidate phenotype vector may be equal to thesimilarity score. The vector distance may be calculated using thefollowing formula (where X₁ represents each dimension defined within thesemantic embedding):

${Distance} = \sqrt{\sum\limits_{i}\left( {X_{i_{target}} - X_{i_{candidate}}} \right)^{2}}$In such embodiments, the lower the value of the similarity score, themore similar two phenotypes may be.

In other embodiments, the value of the similarity score may correspondto the inverse of the magnitude of the similarity score vector. Saidanother way, the inverse of the distance between the target phenotypevector and the candidate phenotype vector is equal to the similarityscore. In these alternate embodiments, the greater the value of thesimilarity score, the more similar two phenotypes may be. Methods ofcalculating similarity score other than distance and inverse distanceare also possible (e.g., a cosine similarity may be calculated todetermine similarity score).

FIG. 3B is a graphical illustration 310 of a target phenotype and athreshold similarity score, according to example embodiments. The targetphenotype may correspond to a target image of a target biological cell.For the target phenotype, the values of each of the dimensions may befound by analyzing the target image using the machine-learned, deepmetric network. Further, the values of each of the dimensionsillustrated in FIG. 3B may be normalized or un-normalized. The targetphenotype is depicted as a vector, having a respective first dimension(X₁) value and a respective second dimension (X₂) value.

Illustrated as a circle in FIG. 3B is a threshold similarity score. Thethreshold similarity score may define a maximum distance at which otherphenotypes are considered to be substantially the same as the targetphenotype. In some embodiments (e.g., embodiments where similarity scoreis defined as the inverse of the distance between the target phenotypeand another phenotype), the threshold similarity score may define aminimum similarity score value at which other phenotypes are consideredto be substantially the same as the target phenotype.

For example, if the target phenotype is an unhealthy phenotype, anycandidate phenotype with a combination of values for each of thedimensions such that a vector representing the candidate phenotyperesides within the circle defined by the threshold similarity score maybe considered an unhealthy phenotype. Similarly, if the target phenotypeis a healthy phenotype, any candidate phenotype with a combination ofvalues for each of the dimensions such that a vector representing thecandidate phenotype resides within the circle defined by the thresholdsimilarity score may be considered a healthy phenotype. In alternateembodiments, where the semantic embeddings define a multi-dimensionalspace having n-dimensions (rather than two, as illustrated in FIG. 3B),the threshold similarity score may correspond to an n-dimensionalsurface. For example, in semantic embeddings having three dimensions,the threshold similarity score may correspond to a spherical shell, asopposed to a circle.

FIG. 3C is a plot 320 of patient stratification, according to exampleembodiments. The plot 320 may illustrate semantic embeddingscorresponding to a plurality of candidate phenotypes 322 (e.g., eachcorresponding to a candidate image of a candidate biological cell). Onlyone of the candidate phenotypes 322 is labelled in FIG. 3C, in order toavoid cluttering the figure. Each of the candidate phenotypes 322 in theplot 320 may have values corresponding to a plurality of dimensions thatrepresent the respective semantic embedding. For example, the candidatephenotypes 322 in FIG. 3C may each have a value for a first dimension(X₁) and a second dimension (X₂). In alternate embodiments, semanticembeddings may include fewer than two dimensions or greater than twodimensions.

As illustrated in FIG. 3C, each of the candidate phenotypes 322 may fallwithin a phenotypic stratum 323 (the surfaces dividing the phenotypicstrata 323 have been labeled in FIG. 3C, rather than the phenotypicstrata themselves, which are the areas enclosed by the dividingsurfaces). In alternate embodiments, only a subset of the candidatephenotypes 322 may fall within a phenotypic stratum 323 (e.g., somecandidate phenotypes 322 may be outside of all phenotypic strata 323bounds). In still other embodiments, one or more of the candidatephenotypes 322 may fall within multiple phenotypic strata 323 (asopposed to a single phenotypic stratum 323). In such embodiments, someof the phenotypic strata 323 may overlap. Also as illustrated in FIG.3C, each of the phenotypic strata 323 may correspond to a contoursurface relative to a target phenotype 321. In some embodiments, each ofthe contour surfaces corresponding to the phenotypic strata 323 may bespaced equidistantly from the target phenotype 321 (e.g., the contoursurfaces may be represented by circles in two-dimensional embodiments orby spheres in three-dimensional embodiments). Alternatively, the contoursurfaces may instead be shaped amorphously. Further, in someembodiments, each of the phenotypic strata 323 may correspond todifferent diseases, different states or phases of a single disease, orhealthy states vs. diseased states.

Similar to FIG. 3C, FIG. 3D is a plot 330 of patient stratification thatincludes a target phenotype 331, candidate phenotypes 332, andphenotypic strata 333 (the surfaces dividing the phenotypic strata 333have been labeled in FIG. 3D, rather than the phenotypic stratathemselves, which are the areas between the dividing surfaces). Only oneof the phenotypes 332 is labelled in FIG. 3D, in order to avoidcluttering the figure. Unlike the plot 320 of FIG. 3C, however, thephenotypic strata 333 do not correspond to contour surfaces relative tothe target phenotype 331. As illustrated, the phenotypic strata 333 aredefined by surfaces (e.g., which are defined by linear equations) thatdepend on values of the dimensions. As in FIG. 3D, in some embodiments,surfaces defining the phenotypic strata 333 may be non-intersecting. Forexample, the phenotypic strata 333 may be defined by surfaces that areparallel to one another (e.g., in the plot 330 of FIG. 3D, thephenotypic strata 333 are defined by parallel lines). Also asillustrated, the phenotypic strata 333 may not be defined by closedsurfaces. Additionally or alternatively, phenotypic strata 333 may bedefined based on groupings of candidate phenotypes 332. For example, thedensities of candidate phenotypes 332 in the left-most phenotypic strata333 and in the right-most phenotypic strata 333 are higher than thedensity of candidate phenotypes 332 in the middle phenotypic strata 333.

In some embodiments, the target phenotype 331 may not be used in thedefinition of the phenotypic strata 333. For example, the targetphenotype 331 may be included simply as a reference point. For example,the target phenotype 331 may be a selected candidate phenotype 332 thatis designated as the target phenotype 331 (e.g., the candidate phenotype332 may be selected as the target phenotype 331 randomly or because itis the centermost candidate phenotype 332 in all dimensions). In yetother embodiments, there may be no target phenotype 331 at all (e.g.,only a series of candidate phenotypes 332 grouped by phenotypic strata333).

Similar to FIGS. 3C and 3D, FIG. 3E is a plot 340 of patientstratification. The plot 340 includes a plurality of candidatephenotypes 342 and a plurality of phenotype strata 343 (the surfacesdividing the phenotypic strata 343 have been labeled in FIG. 3E, ratherthan the phenotypic strata themselves, which are the areas enclosed bythe dividing surfaces). In the plot 340 of FIG. 3E, there is no targetphenotype present. Only one of the candidate phenotypes 342 is labelledin FIG. 3E, in order to avoid cluttering the figure. As described above,the phenotypic strata 343 may be defined based on density variations ofthe candidate phenotypes 342. For example, based on such densityvariations, it may be inferred that regions of high densities ofcandidate phenotypes 342 correspond to a single phenotypic stratum 343.Alternatively, the phenotypic strata 343 may be defined based on variousclustering techniques (e.g., spectral clustering, K-nearest neighborsclustering, machine-learned clustering, or according to a binomialclassifier). As illustrated, the phenotypic strata 343 may correspond tosurfaces of various shapes (e.g., circles, triangles, squares, andrectangles in two-dimensions and cubes, cylinders, cones, rectangularprisms, and pyramids in three-dimensions) and various sizes. Also asillustrated, the scale of a phenotypic stratum 343 in a first dimension(X₁) need not be the same as the scale of a phenotypic stratum 343 in asecond dimension (X₂).

Similar to FIGS. 3C-3E, FIG. 3F is a plot 350 of patient stratification.The plot 350 includes a plurality of candidate phenotypes 352. Only oneof the candidate phenotypes 352 is labelled in FIG. 3F, in order toavoid cluttering the figure. The plot 350 also includes phenotypicstrata 353 (the surfaces dividing the phenotypic strata 353 have beenlabeled in FIG. 3F, rather than the phenotypic strata themselves, whichare the areas between the dividing surfaces). Further, the phenotypicstrata 353 defined in FIG. 3F are only defined by their coordinatevalues in one dimension (e.g., only in X₁, because the phenotypic strata353 are defined by vertical lines), rather than their coordinate valuesin a plurality of dimensions (e.g., as in FIG. 3D where the phenotypicstrata 333 are defined by non-vertical, non-horizontal lines). In someembodiments, for example, the value of a given dimension (e.g., X₁) maydirectly correlate to the state of a disease (e.g., stage one or stagetwo).

FIG. 3G is a plot 370 of patient stratification for a series ofcandidate phenotypes 372. Only one of the candidate phenotypes 372 islabelled in FIG. 3G, in order to avoid cluttering the figure. Thecandidate phenotypes 372 may be divided into a plurality of phenotypicstrata 373 (the surfaces dividing the phenotypic strata 373 have beenlabeled in FIG. 3D, rather than the phenotypic strata themselves, whichare the areas between the dividing surfaces). The phenotypic strata 373may be defined based on densities of the candidate phenotypes 372 in theplot 370.

Further illustrated in FIG. 3G are gradients 374. Each gradient 374indicates the direction of greatest change (e.g., for each dimension (X₁and X₂)) between adjacent phenotypic strata 373. In some embodiments,there may be gradients 374 even without identified phenotypic strata373. For example, a gradient may correspond to a greatest rate ofdensity change from a given point on the plot 370, even if phenotypicstrata 373 have not been identified. Genetic mutations or applicationsof candidate treatment compounds, for example, may move a cell (e.g.,along a gradient 374) from having a candidate phenotype 372 in onephenotypic stratum 373 to having a candidate phenotype 372 in anotherphenotypic stratum 373. Determining which candidate treatment compoundsor genetic mutations will produce a movement of a biological cell alonga gradient 374 (e.g., by reducing or increasing the values of one ormore dimensions of the respective semantic embedding) may result indiscovering a cure for a disease or a palliative treatment for adisease. Alternatively, discovering which genetic mutation will producea movement of a biological cell along a gradient 374 may result indiscovering a cause of a disease state or disease type.

Additionally or alternatively, candidate treatment compounds and geneticmutations may cause a biological cell to change dimensional values(e.g., change X₁ value and/or X₂ value), but not along a gradient. Forexample, applying a candidate treatment compound may result in abiological cell increasing or decreasing its values along one dimension(or a superposition of dimensions) in the respective semantic embedding.This can be used in the treatment or diagnosis of disease. For example,it may be discovered that a first candidate treatment compound increasesa first dimension (X₁) value by 10 units/dose and a second candidatetreatment compound increases a second dimension (X₂) value by 10units/dose. If it is then desired to increase a candidate biologicalcell's first dimension (X₁) value by 20 units and to increase thecandidate biological cell's second dimension (X₂) value by 30 units(e.g., to take the cell from an “unhealthy” phenotypic stratum 373 to a“healthy” phenotypic stratum 373), two doses of the first candidatetreatment compound and three doses of the second candidate treatmentcompound may be applied to the candidate biological cell.

FIG. 3H is an illustration of a topology enforced during training of amachine-learned, deep metric network model, according to exampleembodiments. The topology may be a ring topology 380, as illustrated.Unlike any topologies in FIGS. 3A-3G (e.g., any shapes used to definephenotypic strata), the ring topology 380 may be enforced duringtraining of the machine-learned, deep metric network (e.g., a constrainton the loss function may be imposed such that, during the training ofthe machine-learned, deep metric network model, a projection into a ringshape would result). For example, during training of themachine-learned, deep metric network, a constraint may be applied thatindicates that a specific topology is naturally occurring withinsubjects of the type being studied, and should therefore be reflected insemantic embeddings generated by the machined-learned, deep metricnetwork for subjects of the type being studied. The ring topology 380may be defined by an inner-bounding ring 382 and an outer-bounding ring383, with all semantic embeddings generated for the correspondingsubjects being constrained such that they have dimensional values lyingbetween the inner-bounding ring 382 and the outer-bounding ring 383.

As an example, the ring topology 380 illustrated in FIG. 3H maycorrespond to dimensional values within sematic embeddings of differentmalaria parasites 381 within a malaria population. Because malariaexhibits a lifecycle (e.g., an erythrocytic cycle), a ring topology 380may correspond to different stages in the lifecycle. For example,different regions of the ring topology 380 may correspond to differentcombinations of values of a first dimension (X₁) and a second dimension(X₂). Semantic embeddings corresponding to different malaria parasites381 generated using the machine-learned, deep metric network may begrouped based on their respective first dimension (X₁) and seconddimension (X₂) values (e.g., into six stages of the erythrocytic cycle).For example, the semantic embeddings may be grouped into an erythrocytestage 384, an immature trophozoite stage 385, a mature trophozoite stage386, an erythrocytic schizont stage 387, a ruptured schizont stage 388,and a merozoite stage 389. Hence, by analyzing the combination of valuesfor the first dimension (X₁) and the second dimension (X₂) for anysemantic embedding generated for a respective malaria parasite 381 usingthe machine-learned, deep metric network, the stage of the erythrocyticcycle presently exhibited by the respective malaria parasite 381 can bedetermined.

It is understood that, while the ring topology 380 is depicted astwo-dimensional in FIG. 3H, semantic embeddings generated using themachine-learned, deep metric model may be greater than two-dimensional,thereby resulting in topology that is greater than two-dimensional. Forexample, the ring topology 380 could instead be a toroidal topology inembodiments where the semantic embeddings are three-dimensional. Instill other embodiments, where the semantic embeddings generated aren-dimensional, the ring topology 380 may instead be an n-dimensionalring.

In alternate embodiments (e.g., embodiments where different types ofsubjects are being perturbed, studied, and for which semantic embeddingsare being generated), the topology may be a shape other than a ring.Some example alternate topologies are illustrated in FIGS. 3I-3L. It isunderstood that, while FIGS. 3I-3L illustrate two-dimensional topologiesfor two-dimensional semantic embeddings, other-dimensional topologiesare possible for other-dimensional semantic embeddings (e.g.,three-dimensional topology for three-dimensional semantic embeddings,two-dimensional topology for three-dimensional semantic embeddings,two-dimensional topology for four-dimensional semantic embeddings,three-dimensional topology for four-dimensional semantic embeddings,four-dimensional topology for four-dimensional semantic embeddings,n-dimensional topology for n-dimensional semantic embeddings,(n−1)-dimensional topology for n-dimensional semantic embeddings, etc.).

FIG. 3I is an illustration of a topology enforced during training of amachine-learned, deep metric network, according to example embodiments.The topology may be a linear topology 391, as illustrated. In someembodiments, during training of the machine-learned, deep metricnetwork, a linear topology 391 may attempt to be enforced. Then, only ifthe linear topology 391 is found to be unenforceable based on thetraining data, may other types of topologies attempt to be enforced. Thelinear topology 391 may have upper and lower bounds that are definedwhen training the machine-learned, deep metric network, for example.Depending on a corresponding semantic embedding's location along thelinear topology 391 (e.g., depending on values of the first dimension(X₁) and the second dimension (X₂)), subjects 390 may exhibit certainqualities (e.g., may exhibit qualities inherently or may correspond tocertain perturbations applied to the respective subject 390). Further,the slope and/or intercepts of upper and/or lower bounds may of thelinear topology 391 may correspond to features of the subjects 390extracted during training of the machine-learned, deep metric network.

FIG. 3J is an illustration of a topology enforced during training of amachine-learned, deep metric network model, according to exampleembodiments. The topology may be an elliptical topology 392, asillustrated. The elliptical topology 392 may be enforced when trainingthe machine-learned, deep metric network, for example. Depending on acorresponding semantic embedding's location within (or in otherembodiments, outside of) the elliptical topology 392 (e.g., depending onvalues of the first dimension (X₁) and the second dimension (X₂)),subjects 390 may exhibit certain qualities (e.g., may exhibit qualitiesinherently or may correspond to certain perturbations applied to therespective subject 390). Further, the foci and/or major/minor axes ofthe elliptical topology 392 may correspond to features of the subjects390 extracted during training of the machine-learned, deep metricnetwork.

FIG. 3K is an illustration of a topology enforced during training of amachine-learned, deep metric network model, according to exampleembodiments. The topology may be a hexagonal topology 393, asillustrated. The hexagonal topology 393 may be enforced when trainingthe machine-learned, deep metric network, for example. Depending on acorresponding semantic embedding's location within (or in otherembodiments, outside of) the hexagonal topology 393 (e.g., depending onvalues of the first dimension (X₁) and the second dimension (X₂)),subjects 390 may exhibit certain qualities (e.g., may exhibit qualitiesinherently or may correspond to certain perturbations applied to therespective subject 390). Further, the interior angles and/or sidelengths of the hexagonal topology 393 may correspond to features of thesubjects 390 extracted during training of the machine-learned, deepmetric network.

FIG. 3L is an illustration of a topology enforced during training of amachine-learned, deep metric network model, according to exampleembodiments. The topology may be a cruciform topology 394, asillustrated. The cruciform topology 394 may be enforced when trainingthe machine-learned, deep metric network, for example. Depending on acorresponding semantic embedding's location within (or in otherembodiments, outside of) the cruciform topology 394 (e.g., depending onvalues of the first dimension (X₁) and the second dimension (X₂)),subjects 390 may exhibit certain qualities (e.g., may exhibit qualitiesinherently or may correspond to certain perturbations applied to therespective subject 390). Further, the side lengths and/or location ofintersection of the arms of the cruciform topology 394 may correspond tofeatures of the subjects 390 extracted during training of themachine-learned, deep metric network.

In other embodiments, additional or alternative topologies may also beenforced during training of the machine-learned, deep metric network.For example, circular topologies, polygonal topologies, sphericaltopologies, ellipsoidal topologies, cylindrical topologies, conicaltopologies, hyperboloidal topologies, paraboloidal topologies, toroidaltopologies, pyramidal topologies, polyhedral topologies, and/ortopologies defined in a dimensional space having greater than threedimensions may also be enforced during training of the machine-learned,deep metric network. Still other topologies are also possible.

As described above, identification of a phenotypic stratum or a positionrelative to a given topology of a semantic embedding corresponding to avisual representation may provide information regarding thecorresponding perturbed subject in the visual representation. Forexample, if the corresponding semantic embedding has values for itsrespective dimensions that place it within a given phenotypic stratum orwithin a given portion of a topology (e.g., a given portion of a ringtopology corresponding to a given stage within a cycle), the perturbedsubject may be classified as belonging to one or more groups (e.g., anunhealthy phenotype group or an erythrocytic schizont stage 387 group).

Additionally or alternatively, in some embodiments (e.g., embodimentshaving various types of subjects, biological and/or non-biological),classifying semantic embeddings/visual representations (and, therefore,ultimately classifying corresponding subjects), may include performingvector arithmetic on the semantic embeddings to determine to which ofone or more groups the respective semantic embeddings/visualrepresentations belong. Similar to above, as described with respect tosimilarity scores in FIGS. 3A and 3B, performing vector arithmetic mayinclude determining a vector distance between a given point inn-dimensional embedding space and a respective semantic embedding.Further, performing vector arithmetic may include scaling, normalizing,translating, projecting onto a predetermined plane or axis, etc. one ormore dimensions of a given semantic embedding to determine to which ofone or more groups the semantic embedding belongs. The possible groupsto which a semantic embedding/visual representation may belong may beenumerated in a list of groups, for example.

In still other embodiments, classifying the visualrepresentations/semantic embeddings into one or more groups may includeapplying a classification model (e.g., a machine-learned classificationmodel) to each of the semantic embeddings. The classification model maybe trained to specifically pinpoint responses to any appliedperturbation and/or to disregard any statistical variations inherentlypresent across a population of subjects. For example, the classificationmodel may be trained to ignore naturally occurring variations acrossbiological cell subjects and to instead zero in on those variations thatare induced by applied perturbations (e.g., applied candidate treatmentcompounds).

Additionally or alternatively, in some embodiments, the classificationmodel may be trained using a human-in-the-loop procedure. Such ahuman-in-the-loop procedure may identify which variations (e.g., amongvisual representation training data and/or among generated semanticembeddings) correspond to notable differences (e.g., differences inducedby applied perturbations) and which variations are attributable torandomness (e.g., naturally occurring variations and/or measurementnoise). In still other embodiments, the classification model may betrained using isolation forests (e.g., based on an assumption thatperturbation-induced anomalies should not have the same path lengthsthrough random forests as ordinary variations) to identify whichvariations correspond to perturbation-induced differences and whichvariations are attributable to randomness. Additional or alternativetypes of classification models may also be used.

Based on the one or more groups into which a semantic embedding/visualrepresentation is classified, one or more perturbations applied to thecorresponding subject may be determined. For example, based on values ofparticular dimensions of a semantic embedding corresponding to a visualrepresentation of a subject, the semantic embedding/visualrepresentation may be classified into a certain group (e.g., because asemantic embedding has a value for a first dimension (X₁) of 27.2 and avalue for a second dimension (X₂) of 18.6, the semantic embedding maycorrespond to an erythrocyte stage 384 of an erythrocytic cycle). Thus,the subject represented in the visual representation might alsocorrespond to the certain group (e.g., the subject may be a malariaparasite in the erythrocyte stage 384). Using the group to which thesubject belongs, effects, if any, of any perturbations applied to thesubject can be determined. For example, if a specific wavelength oflight at a certain intensity illuminated a subject, the effect of such aperturbation may be reflected in the group to which the subject belongs.By observing various perturbations applied to a variety of subjectswithin a population and the corresponding groups to which the visualrepresentations/semantic embeddings of the subjects are assignedthereafter, a preferred perturbation from among the variousperturbations may be identified. For example, if an erythrocyte stage384 of an erythrocytic cycle is preferred for clinical reasons, anapplied perturbation that promotes membership in the erythrocyte stage384 among subjects to which the perturbation was applied may bepreferred. Other perturbations that promote membership in non-desiredgroups may be disfavored.

FIG. 4A is a tabular illustration of an un-normalized data set 400having a target phenotype and six candidate phenotypes, according toexample embodiments. The use of a single target phenotype and sixcandidate phenotypes is purely by way of example. In various alternateembodiments, there may be greater or fewer target phenotypes and/orcandidate phenotypes.

The target phenotype may correspond to a healthy phenotype or anunhealthy phenotype of a target biological cell of a target image, invarious embodiments. In other embodiments, the target phenotype maycorrespond to a known disease state or mutation type. Alternatively, insome embodiments, the target phenotype may be defined based on optimizedand/or desired values for each of the dimensions, as opposed to anactual target image of a target biological cell. For example, if Y₁corresponded to a cellular size, the target phenotype may have a valueof Y₁ such that a corresponding biological cell having the targetphenotype has a specified surface area (e.g., 50 μm²).

Further, the candidate phenotypes may correspond to candidate images ofcandidate biological cells acquired from various anatomical regions of apatient during a biopsy, treated with various concentrations of acandidate treatment compound, and/or treated with various candidatetreatment compounds. The candidate images of such candidate biologicalcells may have been recorded from images of a multi-well sample plate.As illustrated, each phenotype in the un-normalized data set 400 mayhave values corresponding to various dimensions (e.g., Y₁ and Y₂) of asemantic embedding. In some embodiments, the various dimensions maycorrespond to human interpretable or non-human interpretable features ofthe semantic embeddings.

FIG. 4B is a graphical illustration 402 of the un-normalized data set400 having the target phenotype and the six candidate phenotypes (asillustrated in FIG. 4A), according to example embodiments. Asillustrated, each of the target phenotype and the candidate phenotypesare represented as vectors in the two-dimensional space defined by theorthogonal dimensions Y₂ and Y₁. The phenotypes in FIG. 4B that arenearer to one another (i.e., those phenotypes whose vector endpoints areshorter distances away from one another) may be more similar to oneanother. For example, as illustrated in FIG. 4B, Candidate Phenotype₂may be more similar to Candidate Phenotype₁ than it is to CandidatePhenotype₄. As another example, as illustrated in FIG. 4B, the TargetPhenotype may be more similar to Candidate Phenotype₅ than it is toCandidate Phenotype₃.

FIG. 4C is a tabular illustration 404 of vector distances 405 betweencandidate phenotypes and a target phenotype of the un-normalized dataset 400 illustrated in FIGS. 4A and 4B, according to exampleembodiments. The vector distances 405 may be calculated using theformula described with respect to FIG. 3A, for example. In someembodiments, the vector distances 405 may correspond to the similarityscore between the candidate phenotypes and the target phenotype. Inother embodiments, the vector distances 405 may correspond to theinverse of the similarity score between the candidate phenotypes and thetarget phenotype.

In some embodiments, the vector distances 405 between the targetphenotype and the candidate phenotypes may be ranked (e.g., orderedascendingly or decreasingly) to determine which of the candidatephenotypes is nearest to the target phenotype. Such a ranking may allowfor a determination to be made regarding which of the candidatephenotypes is most similar to the target phenotype. Additionally oralternatively, the candidate phenotypes may be grouped into similaritysets based on one or more threshold distances or threshold similarityscores. For example, using the vector distances 405 illustrated in FIG.4C, the candidate phenotypes may be grouped into three groups (e.g., onegroup whose vector distances to the target phenotype are between 0.00and 3.00, a second group whose vector distances to the target phenotypeare between 3.00 and 6.00, and a third group whose vector distances tothe target phenotype are greater than 6.00). In such an exampleembodiment, there may be two threshold vector distances (one at 3.00 andone at 6.00). In other embodiments there may be greater or fewer numbersof threshold vector distances (e.g., 0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 20,50, 100, etc.).

In some embodiments, distances between candidate phenotypes may becalculated in addition to distances between the candidate phenotypes andthe target phenotype. Such additional distance calculations may allowfor further similarity analysis among candidate phenotypes.

FIG. 4D is a tabular illustration of a normalized data set 406 having atarget phenotype and six candidate phenotypes, according to exampleembodiments. The normalized data set 406 may correspond to theun-normalized data set 400 illustrated in FIGS. 4A-4C afternormalization. As illustrated, the normalized dimensions (Y′₁ and Y′₂)may correspond to the un-normalized dimensions (Y₁ and Y₂) after anormalization that includes scaling Y₁ by a scaling factor of 0.5. Alsoas illustrated, the normalization of the Y₂ components of theun-normalized data set 400 yielded equivalent values for the Y′₂components of the normalized data set 406.

In some embodiments, the values of the second dimension (Y₂) may bemultiplied by a non-unity scaling factor as well, in order to producethe normalized data set. Additionally or alternatively, in alternateembodiments, one or both of the dimensions of the normalized data setmay be shifted by a certain amount with respect to the un-normalizeddata set. In alternate embodiments having additional dimensions (e.g.,64, 128, 192, or 256 dimensions), additional values for additionaldimensions of the target phenotype and/or the candidate phenotypes mayalso be normalized to achieve a normalized data set.

The un-normalized data set 400 may be normalized to achieve thenormalized data set 406 for multiple reasons. In some embodiments, onereason for normalization may be to generate a set of dimensions thathave a mean of zero and a unit-variance (i.e., the normalization mayinclude “whitening”). If all dimensions within a data set exhibit such anormalized quality, they may not be as prone to outliers stronglyinfluencing the similarity scores.

Additionally or alternatively, another reason for normalization may beto account for typical variation within biological cells (e.g., based onthe phenotypes of negative control groups or positive control groups).Because unperturbed biological cell populations may include a range ofvalues across all or most of the dimensions of morphological variation,it may be important to identify those variations that indicatesimilarity between the candidate phenotypes and the target phenotype andthose variations that arise due to common morphological variation.Accounting for typical variation may include finding dimensions ofcommon morphological variation among the candidate phenotypes and thenreducing the effect of “nuisances” (e.g., false positives) bynormalizing those dimensions. Such a normalization process may increaseor reduce the similarity score of candidate phenotypes that arerelatively abnormal with respect to the similarity score of candidatephenotypes that are relatively common. Accounting for typical variationmay include finding the eigenvalues and the eigenvectors of a covariancematrix of the dimensions using PCA, for example. Further, the transformsused during PCA may be applied to the values of the dimensions of thecandidate phenotypes and the target phenotype to further normalize thedimensions.

FIG. 4E is a graphical illustration 408 of the normalized data set 406having the target phenotype and the six candidate phenotypes, accordingto example embodiments. As illustrated, the phenotypes of the normalizeddata set 406 are closer to one another in the Y′₁ direction than thephenotypes of the un-normalized data set 400 are to one another in theY₁ direction. Similar to FIG. 4B, the phenotypes in FIG. 4E that arenearer to one another (i.e., those phenotypes whose vector endpoints areshorter distances away from one another) may be more similar to oneanother.

FIG. 4F is a tabular illustration 410 of vector distances 411 betweencandidate phenotypes and a target phenotype of the normalized data set406 illustrated in FIGS. 4D and 4E, according to example embodiments.Similar to the vector distances 405 illustrated in FIG. 4C, the vectordistances 411 may be calculated using the formula described with respectto FIG. 3A, for example. Also similar to FIG. 4C, the vector distances411 may correspond to the similarity scores or inverses of similarityscores and the vector distances 411 may additionally be ranked orgrouped.

FIG. 5A is an illustration of an unscaled image 510 of multiple cells512, according to example embodiments. As described above, in otherembodiments, other subjects may be analyzed using the processesdescribed herein. For example, fibroblasts, malaria cells, yeast cells,yeast cultures, bacteria, bacterial cultures, fungus, fungal cultures,cancer cells, blood cells, malarial parasites, mitochondria, nuclei,axons, dendrites, induced pluripotent stem cells, biological cells froma given region of an organism, biological cells from a given tissue ofan organism, biological cells from a given organ of an organism,biological cells from a given system of an organism, biological cellensembles, tissues, organs, organoids, biological systems, organisms,groups of organisms, ecosystems, chemical compounds, crystals, metallicglasses, mixtures of metallic salts, semiconductors, metals,dielectrics, graphene, microelectromechanical systems (MEMS), and/ornanoelectromechanical systems (NEMS) may be analyzed using the processesdescribed herein. Other subjects are also possible.

As illustrated in FIG. 5A, each cell 512 may have an identifiablenucleus 514. The cells 512 may include candidate biological cells and/ortarget biological cells. Further, the cells 512 may have healthy and/orunhealthy phenotypes. In addition, the cells 512 may be from differentanatomical regions of a patient, from different patients, treated withdifferent candidate treatment compounds, treated with differentconcentrations of the same candidate treatment compound, treated withdifferent candidate treatment durations, or have a variety of mechanismsof action. Alternatively, the cells 512 in the unscaled image 510 may bein a negative control group or a positive control group.

The unscaled image 510 may be recorded photographically using one ormore cameras (e.g., from above one or more wells of a multi-well sampleplate), in some embodiments. Additionally, the cameras may include oneor more optical filters to observe specific targeted regions of thecells 512. For example, the cells 512, or part of the cells 512, such asthe nuclei 514, may be dyed with a fluorescent compound that fluorescesat a specific wavelength range. The optical filters may be configured tofilter out light outside of the specific wavelength range. In such away, the unscaled image 510 may be an image of only the targeted regionsof the cells 512. In other embodiments, the unscaled image 510 may be acomposite image that includes multiple channels (e.g., 1, 2, 3, 4, or 5channels), each channel corresponding to a different section of thecells 512 and/or a different wavelength range.

The unscaled image 510 may be various sizes and have various aspectratios in various embodiments. For example, the unscaled image 510 maybe a standardized image size (e.g., 640 pixels by 480 pixels, 256 pixelsby 256 pixels, 128 pixels by 128 pixels, or 100 pixels by 100 pixels) orhave a standardized aspect ratio (e.g., width to height ratio of 4:3,1:1, 2:1, 1:2, etc.). Alternatively, the unscaled image 510 may have anirregular image size and/or aspect ratio. For example, the image sizeand/or aspect ratio may depend on the device (e.g., camera orcharge-coupled device, CCD) used to record the unscaled image 510.

In some embodiments, other types of visual representations of a subjectmay be captured and/or created (e.g., visual representations other thanimages like the unscaled image 510). Such visual representations may begenerated using one or more imaging modalities. For example,three-dimensional visual representations or videographic representationsmay be captured of the subject. Further, in various embodiments (e.g.,embodiments having various subjects), various imaging modalities may beused to capture a visual representation of a subject. For example,computed tomography, magnetic resonance imaging, positron emissiontomography, ultrasound, x-ray computed tomography, x-ray diffraction,fluoroscopy, projectional radiography, single-photon emission computedtomography, scintigraphy, elastography, photoacoustic imaging,near-infrared spectroscopy, magnetic particle imaging, optoacousticimaging, diffuse optical tomography, Raman spectroscopy, fluorescentmicroscopy, confocal microscopy, two-photon microscopy, hyperspectralanalysis, transmission microscopy, electromagnetic scanning,differential interference contrast microscopy, multiphoton microscopy,dark-field microscopy, quantitative phase-contrast microscopy,near-field scanning optical microscopy, photo-activated localizationmicroscopy, second harmonic imaging, holography, scanning electronmicroscopy, and/or tunneling electron microscopy may be used to capturea visual representation of a subject. Other imaging modalities orcombinations of imaging modalities are also possible.

In some embodiments, the imaging modality used to generate the visualrepresentation may perform background correction(s). For example, ifphotobleaching of one or more subjects occurs while imaging thesubjects, that effect may be removed by the imaging modality whencapturing the image (e.g., an image sensor used to capture the visualrepresentation can have a coloration filter and/or neutral-densityfilter applied to compensate for the degree to which coloration and/orbrightness would otherwise be affected by photobleaching when capturingthe visual representation). Additionally or alternatively, backgroundcorrection may be performed after obtaining a semantic embedding for thecorresponding visual representation. For example, if a given backgroundeffect is understood to modify the value of a given dimension (e.g., thefirst dimension X₁) of a semantic embedding by a predetermined amount(e.g., increase the value of the first dimension X₁ by an amount of10.0), the given background effect can be detected among semanticembeddings generated for visual representations and/or can be negatedfrom such semantic embeddings (e.g., by subtracting 10.0 from the firstdimension X₁ for the respective semantic embeddings). Other techniquesof accounting for background correction are also possible.

Further, the unscaled image 510 may be received by a computing device toperform image analysis and comparison (e.g., using a machine-learned,deep metric network). Prior to performing image analysis, the unsealedimage 510 may be transformed in one or more ways. Alternatively, in someembodiments, the computing device may analyze the unsealed image 510 inthe format it is received, without manipulating or transforming theunsealed image 510 (e.g., because the unsealed image 510 is already anappropriate size/scale for interpretation using a machine-learned, deepmetric network model or because the machine-learned, deep metric networkmodel can be used to interpret images of any size or scale).

FIG. 5B is an illustration of a scaled image 520 of multiple cells 522,according to example embodiments. As illustrated, each cell 522 may havean identifiable nucleus 524. The scaled image 520 may be a scaledversion of the unsealed image 510, for example. As illustrated, theunsealed image 510 of FIG. 5A may be transformed into the scaled image520 of FIG. 5B by scaling the horizontal direction of the image by afactor between 0.0 and 1.0 (e.g., 0.5). The scaled image 520 may have anappropriate size (e.g., 640 pixels by 480 pixels, 256 pixels by 256pixels, 128 pixels by 128 pixels, or 100 pixels by 100 pixels) or aspectratio (e.g., width to height ratio of 4:3, 1:1, 2:1, 1:2, etc.) to beinterpretable using the machine-learned, deep metric network.

The scaling illustrated in FIG. 5B may be one of a number oftransformations performed on the unsealed image 510 prior to analysis,by a computing device, of the image using a machine-learned, deep metricnetwork model.

FIG. 5C is an illustration of a single-cell selection process 530 in animage, according to example embodiments. The single-cell selectionprocess 530 may be an additional method used to transform an unsealedimage (e.g., the unsealed image 510 illustrated in FIG. 5A) prior toanalysis using a machine-learned, deep metric network model. The imageoriginally received by a computing device may include multiplebiological cells 532. Each cell may have identifiable features, such asa nucleus 534, for example. As illustrated, the single-cell selectionprocess 530 may include determining a center 536 of one of the cells532. In embodiments where multiple cells 532 are included in the image(e.g., the embodiment illustrated in FIG. 5C), the single cell that isselected out of the group of cells 532 may be selected at random.Alternatively, the cell closest to a certain location of the image maybe selected (e.g., the top-leftmost cell, the top-rightmost cell, thebottom-leftmost cell, the bottom-rightmost cell, or the cell closest tothe center of the image). In other embodiments, a user of a computingdevice may manually select one of the cells 532.

Determining the center 536 of one of the cells 532 may includedetermining the location of the nucleus 534 of the respective cell 532.Additionally or alternatively, determining the center 536 of one of thecells 532 may include determining a horizontal center and/or a verticalcenter of the nucleus 534. Finding the nucleus 534 and/or horizontaland/or vertical centers of the nucleus 534 may include a computingdevice scanning the image for regions of a specific color and/or shapethat corresponds to the shape of the nucleus 534. The color may bedefined by a dye that targets the nucleus 534 (e.g.,4′,6-diamidino-2-phenylindole, DAPI, which targets adenine-thymine pairswithin deoxyribonucleic acid, DNA), for example. Determining thelocation of the nucleus 534 and/or the center 536 of the cell 532 may becompleted using a computing device performing alternative imageprocessing techniques in alternate embodiments.

Further, the single-cell selection process 530 may include selecting aregion 538 of the image surrounding the center 536 of the nucleus 534.The region may be selected based on a typical shape (e.g., circular,rectangular, or elliptical), size (e.g., 128 pixels by 128 pixels),and/or orientation (e.g., vertical, horizontal, or at a 45 degree anglewith respect to the orientation of the image) of biological cells withinthe image. The typical shape, size, and/or orientation of the biologicalcells may be based on a predetermined type of cell (e.g., skin cell,blood cell, cancer cell, nerve cell, muscle cell, pluripotent stem cell,etc.) within the image and/or a predetermined expected phenotype of thecell (e.g., healthy phenotype, unhealthy phenotype, etc.) within theimage. For example, if the cells in the image were expected to be redblood cells having a healthy phenotype, a size and shape of the selectedregion 538 may be based on typical sizes and shapes of healthy red bloodcells at a magnification level corresponding to the magnification levelused to record the image.

The single-cell selection process 530 may select a region that isslightly larger or smaller than a region of the image occupied by onecell (e.g., if the expected size/shape/orientation of the cell does notmatch the actual size/shape/orientation of the cell being analyzed). Forexample, the region 538 selected in FIG. 5C encompasses a regionslightly larger than the cell 532, thereby also encompassing portions ofneighboring cells. Alternatively, in some embodiments, the regionselected by the process may intentionally include multiple cells. Forexample, the shape and size of the region may be based on typical sizesand shapes of regions having multiple cells (e.g., 2, 3, 4, 5, 10, 20,etc. cells). Further, the process for selecting a region having multiplecells may include selecting multiple nuclei or finding the horizontaland/or vertical centers of multiple nuclei.

FIG. 5D is an illustration of a single-cell image 540, according toexample embodiments. The single-cell image 540 may have been generatedby performing the single-cell selection process 530 illustrated in FIG.5C, for example. Additionally, the single-cell image 540 may beextracted from an unscaled image (e.g., the unscaled image 510illustrated in FIG. 5A) or a transformed version of an unscaled image(e.g., the scaled image 520 illustrated in FIG. 5B). As illustrated, thesingle-cell image 540 may include a cell 542 having a nucleus 544. Thecell 542 and the nucleus 544 may be scaled or unscaled versions of thecell 532 and the nucleus 534 illustrated in FIG. 5C, for example.Further, the single-cell image 540 may be compared with othersingle-cell images using a machine-learned, deep metric network model inorder to determine phenotypic similarity, in some embodiments.

The single-cell selection process 530 illustrated in FIG. 5C is anexample of a broad class of techniques referred to as “patch selectiontechniques” used to identify subsets of images (or subsets of othertypes of visual representations) upon which image analysis (e.g.,generation of a corresponding semantic embedding using themachine-learned, deep metric network) will be performed. FIG. 5E is anillustration of another patch selection technique 550, according toexample embodiments. The patch selection technique 550 may be used toselect a patch 552 of the visual representation (e.g., the image) foranalysis using the machine-learned, deep metric network model.

The patch selection technique 550, itself, may be performed using amachine-learned process, in some embodiments. For example, the patchselection technique 550 may be performed using a machine-learned processthat includes an attention mechanism used to define regions of thevisual representation that are of threshold interest level to beselected. For example, as illustrated, the birds in the patch 552 maycorrespond to types of subjects to be analyzed using semanticembeddings, so the attention mechanism may be trained to only selectthose regions of the visual representation that include subjects. Assuch, the attention mechanism may be used to define which regions of apreliminary visual representation are of threshold interest level to beselected by the patch selection. In other embodiments, the patchselection technique 550 may be performed using a set of heuristics(e.g., hard-coded rules to select the location of highest intensitywithin an image; to select a center-region of an image; to select thelocation of an image with the largest red, green, and/or blue value;etc.).

In embodiments where the visual representations are n-dimensional visualrepresentations, the selected patch 552 may, likewise, be n-dimensional.Further, in embodiments where the visual representations arevideographic representations, the selected patch 552 may includemultiple regions of multiple frames of the videographic representation.The regions in the various frames may not necessarily be the samegeometric regions of each frame. For example, if a subject of interest(e.g., pedestrian) is moving through a scene of the videographicrepresentation, the regions of the various frames may move geometricallyacross the scene as well to track the motion of the subject.

FIG. 6A is an illustration of a composite scientific image 600,according to example embodiments. The composite scientific image 600 mayinclude one or multiple channels (e.g., 2, 3, 4, 5, 10, 15, or 20channels) in various embodiments. The composite scientific image 600illustrated in FIG. 6A includes a biological cell 601 having multipleorganelles. As illustrated, the composite scientific image 600 of thecell 601 includes a golgi body 602, mitochondria 604, a nucleus 606(e.g., contain DNA), ribosomes 608, and vacuoles 609. In alternateembodiments, the composite scientific image may show additional oralternative regions of the cell (e.g., a nucleolus, a cytoskeleton, acellular membrane, an endoplasmic reticulum, a vesicle, a lysosome, or acentrosome).

The composite scientific image 600 may be a target image of a targetbiological cell having a target phenotype, for example. In someembodiments, target images may include pixel data. Additionally oralternatively, target images may include an image identification (ID)and a reference. In alternate embodiments, the composite scientificimage 600 may be a candidate image of a candidate biological cell havinga candidate phenotype. Candidate images, similarly, may include pixeldata and/or an image ID/reference. In some embodiments, the compositescientific image 600 may be a scaled or cropped version of a raw imagerecorded by a camera. Further, the composite scientific image 600,either as a whole or individually by channel, may be compared to otherscientific images using a machine-learned, deep metric network.

Each of the channels may represent different target regions of the cell601 or different target components of the cell 601, in some embodiments.The channels may be separated based on wavelength. For example, a dyemay be used to target different components of the cell 601, and eachchannel may be recorded by one or more cameras with selective filtersthat only record light within a given wavelength band, such that onlythe targeted components emitting light within the given wavelength bandmay be measured. In alternate embodiments, various channels of thecomposite scientific image 600 may be distinguished based on otherfactors. For example, in some embodiments, the composite scientificimage may be defined such that the composite scientific image has threechannels, with the first channel being a region defined to be the topthird of the image, the second channel being a region defined to be themiddle third of the image, and the third channel being a region definedto be the bottom third of the image. Other delineations of channelswithin the composite scientific image are also possible.

FIG. 6B is an illustration of a channel 610 that is part of a scientificimage, according to example embodiments. For example, the channel 610may be a component of the composite scientific image 600 illustrated inFIG. 6A. The channel 610 illustrated in FIG. 6B may only show targetedregions or components of the cell 601 based on a fluorescent compoundthat is excited by an electromagnetic source, a chemical dye, or achemiluminescent compound. The targeted regions or componentsillustrated in FIG. 6B may be those regions that are to be investigatedaccording to a given study, for example. As illustrated, the channel 610of FIG. 6B shows the nucleus 606 of the cell 601.

Similar to FIG. 6B, FIG. 6C is an illustration of a channel 620 that ispart of a scientific image (e.g., the composite scientific image 600illustrated in FIG. 6A), according to example embodiments. Asillustrated, the channel 620 of FIG. 6C shows the mitochondria 604 andthe vacuoles 609 of the cell 601.

Similar to FIG. 6B, FIG. 6D is an illustration of a channel 630 that ispart of a scientific image (e.g., the composite scientific image 600illustrated in FIG. 6A), according to example embodiments. Asillustrated, the channel 630 of FIG. 6D shows the golgi body 602 and theribosomes 608 of the cell 601.

FIG. 7A is an illustration of a process 700, according to exampleembodiments. The process 700 may generate a semantic embedding for acomposite scientific image 702, for example, using a machine-learned,deep metric network model. The semantic embedding may then allow thecomposite scientific image 702 to be compared to other images. Thecomposite scientific image 702 may be a scientific image of one or morecells, for example. Further, the scientific image 702 may be a targetimage of a target biological cell having a target phenotype or acandidate image of a candidate biological cell having a candidatephenotype, in various embodiments. The composite scientific image 702,as illustrated, may be similar to the composite scientific image 600illustrated in FIG. 6A (e.g., may be a scaled version of the compositescientific image 600 illustrated in FIG. 6A).

A composite scientific image may include one or more channels (e.g., 1,2, 3, 4, 5, 10, 15, or 20 channels). By way of example, the compositescientific image 702 illustrated in FIG. 7A includes three channels (afirst channel 710, a second channel 720, and a third channel 730). Eachchannel in a composite scientific image may have a given bit depth(e.g., 4, 8, 16, 32, 64, or 128 bit depth). The bit depth of eachchannel of a composite scientific image may be the same as one anotheror different, depending on embodiment.

One step in the process 700 of obtaining a semantic embedding (e.g., bya computing device) for the composite scientific image 702 may includeseparating the channels within the composite scientific image 702. Insome embodiments, each of the individual channels from the compositescientific image 702 may be individually stored in volatile and/ornon-volatile memory (e.g., in a random access memory, RAM, and/or in aread-only memory, ROM, such as a hard drive).

An additional step of the process 700 of obtaining a semantic embeddingfor the composite scientific image 702 may include obtaining a semanticembedding for each respective channel 710, 720, 730 of the compositescientific image 702. As illustrated, a first semantic embedding 712 maybe obtained that corresponds to the first channel 710, a second semanticembedding 722 may be obtained that corresponds to the second channel720, and a third semantic embedding 732 may be obtained that correspondsto the third channel 730.

The semantic embeddings obtained for each channel may correspond tosemantic embeddings that are interpretable using a machine-learned, deepmetric network. For example, each semantic embedding obtained for eachchannel may include equivalent dimensions to those of themachine-learned, deep metric network that were previously learned usingtraining data (e.g., consumer photographic training data arranged intothree-image sets). In addition to including equivalent dimensions (i.e.,dimensions defining corresponding image qualities), the semanticembeddings obtained for each channel may include an equivalent number ofdimensions to those of the machine-learned, deep metric network model.In some embodiments, the semantic embeddings 712, 722, 732 obtained foreach channel may have 16, 32, 64, 96, or 128 dimensions, for example.

In addition, the process 700 of obtaining a semantic embedding for thecomposite scientific image 702 may include concatenating the firstsemantic embedding 712, the second semantic embedding 722, and the thirdsemantic embedding 732 into a unified semantic embedding 740. In otherembodiments where the composite scientific image 702 includes greater orfewer than three channels, the number of semantic embeddingsconcatenated to form a unified semantic embedding may vary. Because theunified semantic embedding 740 is a composite of multiple single-channelsemantic embeddings, the unified semantic embedding 740 may haveadditional dimensionality. For example, if each of the single-channelsemantic embeddings includes 64 dimensions, the unified semanticembedding 740 may have 192 dimensions (3×64). If instead there were onechannel having a single-channel semantic embedding with 64 dimensions,the unified semantic embedding may have 64 dimensions. Further, if therewere five channels having single-channel semantic embeddings with 96dimensions, the unified semantic embedding may have 480 dimensions(5×96), and so on and so forth.

In some embodiments, additional dimensionality reduction may beperformed. For example, dimensionality reduction may be performedindividually on each of the single-channel semantic embeddings.Additionally or alternatively, dimensionality reduction may be performedon the concatenated semantic embedding. Further, in some embodiments,additional dimensions could be defined by comparing the single-channelsemantic embeddings 712, 722, 732 to one another. Such additionaldimensions may be used to analyze information based on inter-channelrelationships, thus ensuring that such information is not lost.

The composite scientific image 702 illustrated in FIG. 7A may correspondto one possible visual representation of a subject (e.g., a perturbedsubject). Additionally or alternatively, the three channels (the firstchannel 710, the second channel 720, and the third channel 730) mayeach, individually, correspond to other types of visual representationsof the subject (e.g., each captured by different imaging modalities). Inother embodiments, the subject may additionally or alternatively berepresented in other types of visual representations.

In some embodiments, a single-channel semantic embedding for a visualrepresentation of a respective subject (e.g., the first semanticembedding 712, the second semantic embedding 722, or the third semanticembedding 732) or a combined semantic embedding for a composite visualrepresentation of a subject (e.g., the unified semantic embedding 740)may be integrated with side data to enhance the respective semanticembedding and improve data analysis performed using the respectivesemantic embedding. The side data may correspond to one or moresupplemental measurements that were performed on the respectivesubjects. Such side data may be represented in the form of supplementalvisual representations. Additionally or alternatively, side data maycorrespond to additional numerical data to be used in conjunction withthe original visual representation. For example, the side data maycorrespond to information accessible only by supplemental measurements(e.g., information that is not accessible simply by analyzing the visualrepresentation generated of the respective perturbed subject). Invarious embodiments, the side data may include experimental metadata(e.g., information about how the experiment was performed, in whatspatial location the experiment was performed, subject provenance forsubjects of the experiment, or preparation history for the experiment),transcriptomic data, genomic data, proteomic data, metabolomic data,lipidomic data, bulk semiconductor material properties, and/or patientdiagnostic data. Other types of side data are also possible.

FIG. 7B is an illustration of a process, according to exampleembodiments. The process corresponds to an integration of side data withvisual representations to arrive at a combined semantic embedding 760.The side data illustrated in FIG. 7B may be metabolomic data 750, forexample. The metabolomic data 750 may correspond to metabolites inand/or around the biological cell illustrated in the compositescientific image 702, for example. Further, the metabolomic data 750 maycorrespond to repeated measurements of metabolites over a period of timeand/or a single measurement of metabolites at a single point in time.

As illustrated in FIG. 7B, one technique by which the side data (e.g.,the metabolomic data 750) may be integrated with the visualrepresentations includes generating one or more additionalsingle-channel semantic embeddings 752 for the side data (e.g., usingthe machine-learned, deep metric network model) and then concatenatingthe additional single-channel sematic embeddings 752 with thesingle-channel semantic emeddings for each channel of the visualrepresentation. For example, similar to FIG. 7A, the visualrepresentation may include three channels, and, correspondingly, maythen have three single-channel semantic embeddings generated (e.g., thefirst semantic embedding 712, the second semantic embedding 722, and thethird semantic embedding 732). Also as illustrated in FIG. 7A, theconcatenation of the first semantic embedding 712, the second semanticembedding 722, and the third semantic embedding 732 may correspond tothe unified semantic embedding 740. A concatenation of the additionalsingle-channel semantic embedding 752 with the first semantic embedding712, the second semantic embedding 722, and the third semantic embedding732 may result in an integrated multi-channel semantic embedding 760, asillustrated. The integrated multi-channel semantic embedding 760 maycontain more information than the unified semantic embedding 740, forexample, and thereby provide more information about a given subject(e.g., provide more data about similarities and differences amongsubjects when comparing the integrated multi-channel semantic embeddings760 for the respective subjects).

FIG. 7C is an illustration of a process, according to exampleembodiments. In some embodiments, as illustrated in FIG. 7C, in additionto or instead of concatenating single-channel semantic embeddings (as inFIG. 7B), integrating the side data (e.g., the metabolomic data 750)with the visual representations of a subject may include producing ahybrid semantic embedding 770.

Similar to FIG. 7B, the process illustrated in FIG. 7C may includegenerating the additional single-channel semantic embedding 752 (e.g.,using the machine-learned, deep metric network model) from themetabolomic data 750. Then, using the first semantic embedding 712, thesecond semantic embedding 722, and the third semantic embedding 732(e.g., individually or in the form of the unified semantic embedding740, as illustrated) in conjunction with the additional single-channelsemantic embedding 752, the hybrid semantic embedding 770 may begenerated. In other embodiments (e.g., embodiments where the visualrepresentation of the subject only contains a single channel), thehybrid semantic embedding 770 may be generated using the additionalsingle-channel semantic embedding 752 and only one single-channelsemantic embedding generated for the visual representation.

In some embodiments, the hybrid semantic embedding 770 may be generatedusing the machine-learned, deep metric network. In other embodiments,the hybrid semantic embedding 770 may be generated using a differentmachine-learned model (e.g., an autoencoder or a machine-learned modeltrained to combine dimensional values from previously generated semanticembeddings). In various embodiments, the hybrid semantic embedding 770may be single-channel or multi-channel (e.g., but not generated usingconcatenation).

FIG. 7D is an illustration of a process, according to exampleembodiments. In some embodiments, as illustrated in FIG. 7D, in additionto or instead of concatenating single-channel semantic embeddings (as inFIG. 7B) or producing a hybrid semantic embedding 770 using amachine-learned model (as in FIG. 7C), integrating the side data (e.g.,the metabolomic data 750) with the visual representations of a subjectmay include producing a composite semantic embedding 780 by applying amathematical operation to corresponding dimensions of the semanticembeddings generated for the visual representation and the side data.

Similar to FIGS. 7B and 7C, the process illustrated in FIG. 7D mayinclude generating the additional single-channel semantic embedding 752(e.g., using the machine-learned, deep metric network model) from themetabolomic data 750. Then, by applying one or more mathematicaloperations to the generated first semantic embedding 712, the secondsemantic embedding 722, the third semantic embedding 732 (e.g., in theform of the unified semantic embedding 740 or individually, asillustrated), and the additional single-channel semantic embedding 752,the composite semantic embedding 780 may be generated. In otherembodiments (e.g., embodiments where the visual representation of thesubject only contains a single channel), the composite semanticembedding 780 may be generated using the additional single-channelsemantic embedding 752 and only one single-channel semantic embeddinggenerated for the visual representation.

As illustrated, the composite semantic embedding 780 may represent anaverage of the first semantic embedding 712, the second semanticembedding 722, the third semantic embedding 732, and the additionalsingle-channel semantic embedding 752. For example, the value for arespective dimension within the composite semantic embedding 780 may bean average of the values for that respective dimension from each of thefirst semantic embedding 712, the second semantic embedding 722, thethird semantic embedding 732, and the additional single-channel semanticembedding 752. In other embodiments, other mathematical operations maybe used in addition to or instead of averaging. For example, weightedaverages, summations, weighted summations, differences, weighteddifferences, products, weighted products, quotients, weighted quotients,geometric averages, weighted geometric averages, and/or medians may beused to generate the composite semantic embedding 780. Othermathematical operations are also possible.

FIG. 8 is an illustration 800 of anatomical regions of a patient fromwhich candidate biological cells and a target biological cell may beacquired, according to example embodiments. For example, in someembodiments, it may be known that a patient has a disease in onelocation of their body (e.g., lung cancer). During a biopsy, a sample ofcells from the anatomical region with the known disease may be acquired.These cells may thus be indicated to have an unhealthy phenotype and maybe indicated to be target biological cells. Also during the biopsy,samples of cells may be acquired from other anatomical regions of thebody. For example, samples may also be acquired from the brain, theliver, the large intestine, the heart, the stomach, the kidneys, and thesmall intestine. Samples may also be taken from other anatomical regionsof the body in various embodiments (e.g., the skin to study fibroblastsor the blood to study leukocytes). These additional samples may belabelled as candidate biological cells having candidate phenotypes.

Upon retrieving the target biological cells and the candidate biologicalcells, a target image and several candidate images may then be recorded.The images may then be compared by a computing device using amachine-learned, deep metric network, in some embodiments. If any of thecandidate images have a similarity score with the target image that isabove a threshold similarity score, for example, those candidate imagesmay then be determined to correspond to candidate biological cells thathave a similar phenotype to the target phenotype (an unhealthyphenotype, for example). This may indicate, as in the example of lungcancer, that the cancer has metastasized to another anatomical region ofthe patient (e.g., the anatomical region from which the candidatebiological cells having a similar phenotype to the target phenotype wereacquired). The converse is also possible, in alternate embodiments. Forexample, an anatomical region of the patient may include target cellsthat are known to have a healthy phenotype, and a diagnostic may be run(e.g., using a computing device that uses a machine-learned, deep metricnetwork to perform image comparisons) to evaluate whether candidatecells from other anatomical regions also have healthy phenotypes (e.g.,if the candidate cells have a similarity score with the target cellsthat is greater than a threshold similarity score).

In still other embodiments, target cells within certain anatomicalregions of the body may be known to exhibit certain mechanisms of actionor respond to certain stimuli in certain ways, based on their phenotype.Again, candidate cells in other anatomical regions of the body could becompared and contrasted with the target cells, to establish if theirphenotypes are similar to the target phenotype of the target cells.

FIG. 9 is an illustration of a multi-well sample plate 900, according toexample embodiments. The multi-well sample plate 900 may include variousnumbers of rows and columns, in various embodiments. As illustrated inFIG. 9, the multi-well sample plate 900 includes eight rows (row 9A, row9B, row 9C, row 9D, row 9E, row 9F, row 9G, and row 9H) and twelvecolumns (column 901, column 902, column 903, column 904, column 905,column 906, column 907, column 908, column 909, column 910, column 911,and column 912). Other numbers of rows and/or columns may be used inalternate embodiments. The multi-well sample plate 900 may be loadedwith various samples in various wells in order to generate dataanalyzable using a computing device that uses a machine-learned, deepmetric network model, for example.

In one embodiment, one subset of the wells of the multi-well sampleplate 900 (e.g., row 9G) may be loaded with biological cells from anegative control group. This group of cells may be equivalent to a groupof candidate biological cells initially, but may not receive anytreatment. Another subset of the wells of the multi-well sample plate900 (e.g., row 9H) may be loaded with target biological cells having aknown target phenotype (e.g., a healthy phenotype). The remaining rowsof wells of the multi-well sample plate 900 may be loaded with candidatebiological cells.

The candidate biological cells may be known to initially have a givenphenotype (e.g., an unhealthy phenotype). Thereafter, the candidatebiological cells may be treated with various candidate treatmentcompounds at various concentrations. For example, all candidatebiological cells in row 9A may be treated with candidate treatedcompound one, all candidate biological cells in row 9B may be treatedwith candidate treatment compound two, all candidate biological cells inrow 9C may be treated with candidate treatment compound three, and soon. In addition, all candidate biological cells in column 901 (excludingthe negative control biological cells and target biological cells incolumn 901, row 9G and column 901, row 9H, respectively) may be treatedwith the respective candidate treatment compound of their row at aconcentration of 0.1 molar. Similarly, all candidate biological cells incolumn 902 may be treated with the respective candidate treatmentcompound of their row at a concentration of 0.2 molar, and so on for therest of the columns through 1.2 molar.

As an example, row 9C may be treated with candidate treatment compoundthree in concentrations of 0.1 molar in column 901, 0.2 molar in column902, 0.3 molar in column 903, 0.4 molar in column 904, 0.5 molar incolumn 905, 0.6 molar in column 906, 0.7 molar in column 907, 0.8 molarin column 908, 0.9 molar in column 909, 1.0 molar in column 910, 1.1molar in column 911, and 1.2 molar in column 912. As an additionalexample, column 905 may be treated with concentrations of 0.5 molarusing candidate treatment compound one in row 9A, candidate treatmentcompound two in row 9B, candidate treatment compound three in row 9C,candidate treatment compound four in row 9D, candidate treatmentcompound five in row 9E, and candidate treatment compound six in row 9F.

In other embodiments (e.g., embodiments where the subjects are notbiological cells), multi-well sample plates may not be used. Further, adifferent perturbation (e.g., a perturbation other than applying acandidate treatment compound) may be applied to the subjects in otherembodiments. For example, applying a perturbation may includeselectively mixing two or more metal salts (e.g., Iron, Mn, and Mg)using predefined ratios and then heating the metal salts to allow themto oxidize. Visual representations of the oxidized, mixed metal saltsmay then be generated using one or more imaging modalities. For example,colors (e.g., imaged using different wavelengths, either within thevisible spectrum or outside of the visual spectrum) within the visualrepresentation and/or textures within the visual representation maycorrespond to what the chemical composition of the perturbed subject is.Thereafter, the visual representations of the oxidized, mixed metalsalts may be used to generate semantic embeddings and then the semanticembeddings may be compared/analyzed to determine specificcharacteristics of the oxides (e.g., to back out chemical compositionsof the oxides and/or to identify desired characteristics of one or moreoxides, thereby identifying desired ratios of the two or more metalsalts). In embodiments using other types of perturbations, othermaterial properties may additionally or alternatively be analyzed usingsemantic embeddings generated for visual representations of materials(e.g., wetting properties, electrical surface properties, opticalsurface properties, coefficients of friction, hardness, ductility,elasticity, viscosity, porosity, tensile strength, compressive strength,packing fraction, capacitance, inductance, dielectric constant, relativemagnetic permeability, absorbance, reflectivity, transmittance,coefficient of thermal expansion, melting point, boiling point,flammability, emissivity, flash point, specific heat, thermalconductivity, etc. may be analyzed).

In various embodiments (e.g., depending on the subjects to beperturbed), applying a perturbation to a plurality of subjects mayinclude adding one or more small molecule compounds to at least one ofthe plurality of subjects; applying a genetic modification to at leastone of the plurality of subjects; allowing a predetermined amount oftime to elapse for at least one of the plurality of subjects;illuminating at least one of the plurality of subjects with light of apredetermined wavelength; heating or cooling at least one of theplurality of subjects to a predetermined temperature; exposing at leastone of the plurality of subjects to another subject of the plurality ofsubjects; introducing at least one of the plurality of subjects to adifferent environment; and/or applying a predetermined force to one ormore regions of at least one of the plurality of subjects. In otherembodiments, applying a perturbation to a subject may include alternateperturbations or a combination of the above-described perturbations.

Negative control images of the negative control group biological cells,target images of the target biological cells, and candidate images ofthe candidate biological cells may then be recorded of the biologicalcells from the different wells of the multi-well sample plate 900. Thenegative control images, the candidate images, and the target images maythen be compared to one another (e.g., by a computing device using amachine-learned, deep metric network model).

In other embodiments, other delineations may be drawn between sampleswithin different wells of the multi-well sample plate 900. For example,in some embodiments, negative control group samples, candidate samples,or target samples may correspond to various healthy phenotype(s),various unhealthy phenotype(s), various candidate compounds, variouscandidate compound concentrations, various candidate treatmentdurations, various anatomical regions of a single patient, a commonanatomical region across various patients, various anatomical regionsacross various patients, various mechanisms of action (e.g., analyzed byproviding specific wells of the multi-well sample plate 900 with variousinhibitors), and/or various compounds for illuminating specific cellularregions (e.g., fluorescent compounds, chemical dyes, or chemiluminescentcompounds). Other candidate variations for similarity study amongvarious cells are also possible.

For example, in alternate embodiments, one subset of the wells of themulti-well sample plate 900 (e.g., row 9G) may be loaded with biologicalcells from a negative control group. This group of cells may beequivalent to a group of candidate biological cells initially, but maynot receive any treatment and/or genetic modifications (e.g., targetedmutations). Another subset of the wells of the multi-well sample plate900 (e.g., row 9H) may be loaded with a first set of candidatebiological cells (e.g., somatic cells, fibroblasts, or inducedpluripotent stem cells) having a known genetic mutation (e.g., a geneticmutation whose presence prevents the occurrence of a disease, such as bycoding for the production of a particular protein; whose presenceincreases or decreases cellular and organismal growth rate; whosepresence leads to a disorder or disease; etc.). Such a genetic mutationmay be naturally occurring or may be the result of targeted geneticmodifications (e.g., using a clustered regularly interspaced shortpalindromic repeats (CRISPR) screen), in various embodiments. Theremaining rows of wells of the multi-well sample plate 900 may be loadedwith a second set of candidate biological cells (e.g., somatic cells,fibroblasts, or induced pluripotent stem cells).

The first set of candidate biological cells and the second set ofcandidate biological cells may be known to initially have a givenphenotype (e.g., a healthy phenotype). Thereafter, the second set ofcandidate biological cells may be given various candidate treatmentcompounds at various concentrations. For example, all of the second setof candidate biological cells in row 9A may be treated with candidatetreated compound one, all of the second set of candidate biologicalcells in row 9B may be treated with candidate treatment compound two,all of the second set of candidate biological cells in row 9C may betreated with candidate treatment compound three, and so on. In addition,all of the second set of candidate biological cells in column 901(excluding the negative control biological cells and target biologicalcells in column 901, row 9G and column 901, row 9H, respectively) may betreated with the respective candidate treatment compound of their row ata concentration of 0.1 molar. Similarly, all of the second set ofcandidate biological cells in column 902 may be treated with therespective candidate treatment compound of their row at a concentrationof 0.2 molar, and so on for the rest of the columns through 1.2 molar.

As an example, row 9C may be treated with candidate treatment compoundthree in concentrations of 0.1 molar in column 901, 0.2 molar in column902, 0.3 molar in column 903, 0.4 molar in column 904, 0.5 molar incolumn 905, 0.6 molar in column 906, 0.7 molar in column 907, 0.8 molarin column 908, 0.9 molar in column 909, 1.0 molar in column 910, 1.1molar in column 911, and 1.2 molar in column 912. As an additionalexample, column 905 may be treated with concentrations of 0.5 molarusing candidate treatment compound one in row 9A, candidate treatmentcompound two in row 9B, candidate treatment compound three in row 9C,candidate treatment compound four in row 9D, candidate treatmentcompound five in row 9E, and candidate treatment compound six in row 9F.

After treating the second set of candidate biological cells, atime-evolution of the first set of candidate biological cells and thesecond set of candidate biological cells may be monitored. In someembodiments, prior to the time-evolution being monitored, a parasite,bacterium, virus, etc. may be introduced to each of the wells having thefirst set of candidate biological cells and/or the second set ofbiological cells (e.g., to test for a resistance to the parasite,bacterium, virus, etc.). Further, in some embodiments, the first setand/or the second set of candidate biological cells may be infected withone or more diseases (e.g., Parkinson's disease, amyotrophic lateralsclerosis, or Alzheimer's disease). In such embodiments, the negativecontrol biological cells may also be infected with the disease.

Negative control images of the negative control group biological cellsand candidate images of the candidate biological cells may then berecorded of the biological cells from the different wells of themulti-well sample plate 900. The negative control images and thecandidate images may then be compared to one another (e.g., by acomputing device using a machine-learned, deep metric network model).Comparing the candidate images of the first set of candidate biologicalcells to the candidate images of the second set of candidate biologicalcells (e.g., using a machine-learned, deep metric network model) mayallow for an identification of which, if any, of the candidate treatmentcompounds mirrors or mimics the results of the genetic mutations (e.g.,after introduction of the parasite, bacterium, virus, or disease). Suchan identification of one or more candidate treatment compounds thatmimics one or more of the genetic modifications may be based on vectordistances in a multi-dimensional space described by semantic embeddingsbetween candidate images (e.g., between a candidate image recorded ofthe first set of candidate biological cells and a candidate imagerecorded of the second set of candidate biological cells).

As described above with reference to FIG. 3G, identifying molecularmimetics of genetic perturbations (e.g., mutations) may includeidentifying a plurality of phenotypic strata (e.g., phenotypic stratarepresenting various subpopulations). Then, based on the phenotypicstrata, one or more gradients may be determined (e.g., gradients betweenstrata). One or more of these gradients may represent certain geneticperturbations. Additionally or alternatively, one or more of thesegradients may represent an applied candidate treatment compound (e.g.,in a given concentration). If there is a threshold level of similarity(e.g., in slope or intercept) between one of the gradients correspondingto a genetic perturbation and one of the gradients corresponding to anapplied candidate treatment compound, a computing device (e.g., using amachine-learned, deep-metric network) may determine that the respectiveapplied candidate treatment compound mimics the genetic perturbation.Identifying such a mimicking could later be used for diagnosing and/ortreating patients. For example, such an identification may be used inthe development of a therapeutic drug that mimics a geneticmodification.

Alternatively, in some embodiments, it may be determined (e.g., based onsemantic embeddings) that a given genetic mutation corresponds to agiven rate of change (e.g., per unit time) in a first dimension of asemantic embedding. If it is determined that a given candidate treatmentcompound corresponds to a given rate of change (e.g., per unit time) ina first dimension of a semantic embedding that is of thresholdsimilarity to the given rate of change corresponding to the geneticmutation, a computing device (e.g., using a machine-learned, deep-metricnetwork) may determine that the respective candidate treatment compoundmimics the genetic perturbation. In such embodiments, both the geneticperturbation and the application of the respective candidate treatmentcompound may move a biological cell along the first dimension. In otherembodiments, the genetic perturbations and/or the candidate treatmentcompounds may correspond to a superposition of rates of change (e.g.,per unit time) in multiple dimensions of a semantic embedding.

FIG. 10 is an illustration of a method 1000, according to exampleembodiments. In some embodiments, each block may represent a module, asegment, or a portion of program code, which includes one or moreinstructions executable by a processor or computing device forimplementing specific logical operations or steps. The program code maybe stored on any type of computer-readable medium, such as a storagedevice included in a disk or hard drive. The computer-readable mediummay include volatile memory, such as a register memory, a processorcache, and/or a RAM. Additionally or alternatively, thecomputer-readable medium may include non-volatile memory, such assecondary or persistent long-term storage, like ROM, optical or magneticdisks, and compact-disc read-only memory (CD-ROM), for example. Inaddition, one or more blocks in FIG. 10 may represent circuitry that iswired to perform the specific logical operations.

In some embodiments, the method 1000 may include additional blocksoccurring before, in between, or after the blocks illustrated in FIG.10. Further, in some embodiments, one or more of the blocks illustratedin FIG. 10 may be repeated one or more times. In alternate embodiments,the order of various blocks within the method may be rearranged withoutdeparting from the scope of the method.

At block 1002, the method 1000 includes receiving, by a computingdevice, a target image of a target biological cell having a targetphenotype. The target image may include image data (e.g., the pixelvalues used to generate the image) and/or a reference value associatedwith a location of an associated semantic embedding (e.g., a memoryaddress associated with the associated semantic embedding). The targetimage may have been recorded by a camera, for example. Further, in someembodiments, the camera may have one or more associated optical filtersconfigured to allow the transmission of only a range of wavelengths tothe camera. Additionally, the camera may transmit the target image tothe computing device. For example, the camera may communicate with thecomputing device over WiFi (IEEE 802.11 standards), over Bluetooth®, orvia wireline interface (e.g., a universal serial bus, USB, cable).

In alternate embodiments, the computing device may receive the targetimage through communication with another computing device. For example,the computing device may receive the target image from a mobilecomputing device (e.g., a mobile phone equipped with a camera thatrecorded the target image), a tablet computing device, or a personalcomputing device (e.g., a laptop computing device). The computing devicemay receive the target image via an application (app) or throughelectronic mail (email), in various embodiments.

In some embodiments, the target image received by the computing devicemay be accompanied by target image metadata. For example, the targetimage metadata may include when the target image was recorded, thenumber of channels in the target image, the bit-depth of each channel inthe target image, to which wavelength ranges or cellular components eachchannel in the target image corresponds, a predetermined targetphenotype associated with the target cells in the target image, atreatment regimen provided to the target cells in the target image, themechanisms of action occurring in the target cells of the target image,a row of a multi-well sample plate from which the target image wasrecorded, a column of a multi-well sample plate from which the targetimage was recorded, or an anatomical region of a patient from which thetarget cells in the target image were acquired.

At block 1004, the method 1000 includes obtaining, by the computingdevice, a semantic embedding associated with the target image. Thesemantic embedding associated with the target image may be generatedusing a machine-learned, deep metric network model. The machine-learned,deep metric network model may have been previously trained usingconsumer photographic training data. For example, the consumerphotographic training data may include three-image sets (e.g., similarto the three-image set illustrated in FIG. 1). The three-image sets maybe based upon internet search results and selections (e.g., by internetusers). The internet search results and selections may allow thecomputing device to train the machine-learned, deep metric network modelto identify similarities and differences between images. In someembodiments, particular features of images will be separated intodimensions (e.g., 64 dimensions) of the semantic embedding during thetraining of the machine-learned, deep metric network.

Further, in some embodiments, obtaining the semantic embeddingassociated with the target image may include a similar process to theprocess illustrated in FIG. 7A. For example, obtaining the semanticembedding associated with the target image may include retrieving thechannels of the target image, obtaining a semantic embedding for eachchannel of the target image, and concatenating each channel of thetarget image into a single semantic embedding. In some embodiments, eachof the channels may correspond to a predetermined range of wavelengthsdetectable by one or more cameras (e.g., to investigate one or moreregions of the target biological cell). The concatenated semanticembedding may have more dimensions than the single-channel semanticembeddings (e.g., 192 dimensions for a concatenated semantic embeddingof a target image having three channels, each channel having a 64dimension single-channel semantic embedding).

In alternate embodiments, alternate processes of obtaining a semanticembedding associated with the target image may additionally oralternatively be used. For example, an autoencoder may generate asemantic embedding associated with the target image. In otherembodiments, a classification model other than the machine-learned, deepmetric network model may be used to obtain the semantic embedding. Forexample, an output or a hidden layer of another artificial neuralnetwork may generate the semantic embedding for the target image. Instill other embodiments, variations of the machine-learned, deep metricnetwork model trained on three-image sets may be used to obtain thesemantic embedding of the target image. For example, themachine-learned, deep metric network model may be trained with imagesthat more closely resemble the target image and the candidate images(e.g., the model is not trained on consumer photographic query results),such as only scientific images or only images of biological cells.

At block 1006, the method 1000 includes obtaining, by the computingdevice for each of a plurality of candidate images of candidatebiological cells (e.g., each corresponding to a candidate mechanism ofaction) each having a respective candidate phenotype, a semanticembedding associated with the respective candidate image. The candidateimages may include image data (e.g., the pixel values used to generatethe image) and/or a reference value associated with a location of anassociated semantic embedding (e.g., a memory address associated withthe associated semantic embedding). Analogous to block 1002, thecandidate images may have been recorded by a camera. Likewise, in someembodiments, the camera may have one or more associated optical filtersconfigured to allow the transmission of only a range of wavelengths tothe camera. Additionally, the camera may transmit the candidate imagesto the computing device. For example, the camera may communicate withthe computing device over WiFi, over Bluetooth®, or via wirelineinterface (e.g., a USB cable).

In alternate embodiments, the computing device may receive the candidateimages through communication with another computing device. For example,the computing device may receive the candidate images from a mobilecomputing device (e.g., a mobile phone equipped with a camera thatrecorded the candidate image), a tablet computing device, or a personalcomputing device (e.g., a laptop computing device). The computing devicemay receive the candidate images via an app or through email, in variousembodiments.

In some embodiments, the candidate images received by the computingdevice may be accompanied by candidate image metadata. For example, thecandidate image metadata may include when the candidate images wererecorded, the number of channels in the candidate images, the bit-depthof each channel in the candidate images, to which wavelength ranges orcellular components each channel in the candidate images corresponds,treatment regimens provided to the candidate cells in the candidateimages, the mechanisms of action occurring in the candidate cells of thecandidate images, a row of a multi-well sample plate from which thecandidate images were recorded, a column of a multi-well sample platefrom which the candidate images were recorded, or an anatomical regionof a patient from which the candidate cells in the candidate images wereacquired.

Similar to block 1004, the semantic embeddings associated with thecandidate images may be generated using a machine-learned, deep metricnetwork model. The machine-learned, deep metric network model may be thesame model as in block 1004 (i.e., may have been previously trainedusing consumer photographic training data). Also similar to block 1004,in some embodiments, obtaining the semantic embeddings associated withthe candidate images may include a similar process to the processillustrated in FIG. 7A. Also similar to block 1004, alternate processesof obtaining the semantic embeddings associated with the respectivecandidate images may be used (e.g., semantic embeddings generated by anautoencoder or semantic embeddings obtained using a variation on themachine-learned, deep metric network model).

At block 1008, the method 1000 includes determining, by the computingdevice, a similarity score for each candidate image. Determining thesimilarity score for a respective candidate image includes computing, bythe computing device, a vector distance in a multi-dimensional spacedescribed by the semantic embeddings between the respective candidateimage and the target image. The similarity score for each candidateimage represents a degree of similarity between the target phenotype andthe respective candidate phenotype.

In some embodiments, the method 1000 may also include training themachine-learned, deep metric network model. Training themachine-learned, deep metric network model may include receiving, by thecomputing device, a series of three-image sets as training data. Eachthree-image set may include a query image, a positive image, and anegative image. The query image, the positive image, and the negativeimage may be photographic internet search results ranked in comparisonwith another based on selections by internet users, for example. Inaddition, based on the selections by internet users, it is determined(e.g., by the computing device) that a similarity between the queryimage and the positive image is greater than a similarity between thequery image and the negative image. The method 1000 may additionallyinclude refining, by the computing device, the machine-learned, deepmetric network model based on each three-image set to account for imagecomponents of the query image, the positive image, and/or the negativeimage.

Further, in some embodiments, the method 1000 may include receiving, bythe computing device, a plurality of control group images of controlgroup biological cells having control group phenotypes. In suchembodiments, the method 1000 may also include obtaining, by thecomputing device for each control group image, a semantic embeddingassociated with the respective control group image. Further, in suchembodiments, the method 1000 may also include normalizing, by thecomputing device, the semantic embeddings associated with the candidateimages. Normalizing may include computing, by the computing device,eigenvalues and eigenvectors of a covariance matrix defined by thevalues of each dimension of the semantic embeddings associated with thecontrol group images using principal component analysis. Normalizing mayalso include scaling, by the computing device, values of each dimensionof the semantic embeddings associated with the control group images by arespective dimensional scaling factor such that each dimension iszero-centered and has unit variance. Further, normalization may includescaling, by the computing device, values of each corresponding dimensionof the semantic embeddings associated with the candidate images by therespective dimensional scaling factor. Such normalization of thesemantic embeddings associated with the candidate images may negate aninfluence of common morphological variations (e.g., variations incellular size, in nuclear size, in cellular shape, in nuclear shape, innuclear color, in nuclear size relative to cellular size, or in nuclearlocation within a respective cell) among the candidate biological cellson the similarity scores. In some embodiments, normalizing may furtherinclude shifting, by the computing device, values of each correspondingdimension of the semantic embeddings associated with the candidateimages by a dimensional shifting factor.

Even further, in some embodiments, the method 1000 may includeadditional analysis of the similarity scores. As described with respectto FIGS. 4C and 4F, the similarity scores may be ranked to determinewhich candidate image has the greatest similarity score (i.e., themaximum similarity score) among the candidate images. The candidateimage with the greatest similarity score may be determined to be mostsimilar to the target image. Thus, the candidate image with the greatestsimilarity score may be determined to correspond to a candidatebiological cell that has a candidate phenotype that is most similar tothe target phenotype of the target biological cell associated with thetarget image. In some embodiments (e.g., where the target phenotype is ahealthy phenotype), the method 1000 may also include determining, by thecomputing device, a treatment regimen for a patient based on thecandidate biological cell having the candidate phenotype that is mostsimilar to the target phenotype. The treatment regimen determined may bea treatment regimen applied to the candidate biological cellcorresponding to the greatest similarity score, for example.

Additionally or alternatively, the similarity scores may be compared toa single threshold similarity score to determine which of thecorresponding candidate images exhibits a threshold level of similaritywith the target image (e.g., those similarity scores that are greaterthan or equal to the threshold similarity score exhibit the thresholdlevel of similarity with the target image). In still other embodiments,the similarity scores may be grouped into multiple groups of candidateimages based on multiple threshold similarity scores (e.g., defined suchthat each group of candidate images has the same number of candidateimages in the group or defined such that certain sets of similaritycharacteristics between the candidate images and the target image areminimally shared by all members of a group). The candidate images mayadditionally be group in other ways based on their semantic embeddingsthat do not include the calculation of a similarity score with thetarget image. For example, all candidate images having a value ofdimension Z₁ that is greater than ε may be put into a group. In thisway, candidate images having a given characteristic may be placed into acommon group, even if those candidate images do not have comparablesimilarity scores with one another.

In alternate embodiments, the method 1000 may also include scaling, bythe computing device, the target image and each of the candidate imagessuch that an image size of the target image and image sizes of each ofthe candidate images match an image size (e.g., a standardized size)interpretable using the machine-learned, deep metric network model.Scaling the target image and each of the candidate images may includedetermining, by the computing device, a location of a cellular nucleuswithin the respective image. Scaling the target image may also includecropping, by the computing device, the respective image based on arectangular box centered on the cellular nucleus.

FIG. 11 is an illustration of a method 1100, according to exampleembodiments. In some embodiments, various blocks may represent a module,a segment, or a portion of program code, which includes one or moreinstructions executable by a processor or computing device forimplementing specific logical operations or steps. The program code maybe stored on any type of computer-readable medium, such as a storagedevice included in a disk or hard drive. The computer-readable mediummay include volatile memory, such as a register memory, a processorcache, and/or a RAM. Additionally or alternatively, thecomputer-readable medium may include non-volatile memory, such assecondary or persistent long-term storage, like ROM, optical or magneticdisks, and CD-ROM, for example. In addition, one or more blocks in FIG.11 may represent circuitry that is wired to perform the specific logicaloperations.

In some embodiments, the method 1100 may include additional blocksoccurring before, in between, or after the blocks illustrated in FIG.11. Further, in some embodiments, one or more of the blocks illustratedin FIG. 11 may be repeated one or more times. In alternate embodiments,the order of various blocks within the method may be rearranged withoutdeparting from the scope of the method.

At block 1102, the method 1100 includes preparing a multi-well sampleplate with a target biological cell having a target phenotype (e.g., ahealthy phenotype) and candidate biological cells. Preparing themulti-well sample plate may include loading specific numbers,concentrations, or volume of target biological cells and/or candidatebiological cells into wells of the multi-well sample plate. This mayinclude apportioning aliquots of a target sample and/or of a candidatesample into wells of the multi-well sample plate. Further, preparing themulti-well sample plate may include loading the target biologicalcell(s) and the candidate biological cell(s) into specified wells of themulti-well sample plate. For example, the target biological cell(s) maybe loaded into all wells occupying a first column of the multi-wellsample plate. Additionally, the candidate biological cell(s) may beloaded into all wells occupying the remaining columns of the multi-wellsample plate. Other loading schemata are also possible in variousembodiments.

The multi-well sample plate may be a microtiter plate, in some exampleembodiments. The multi-well sample plate may also have a variety ofnumbers of wells in various embodiments. For example, the multi-wellsample plate may have 8, 16, 24, 64, 96, 384, or 1536 wells, in variousembodiments. Other numbers of wells are also possible. In someembodiments, other containers may be used to hold the samples (e.g., thetarget biological cells and/or the candidate biological cells). Forexample, in some embodiments, the samples may be held in centrifugetubes, beakers, test tubes, petri dishes, vials, flasks, graduatedcylinders, burets, and/or microscope slides. Alternative samplecontainers are also possible in various embodiments.

At block 1104, the method 1100 includes applying a variety of candidatetreatment regimens to each of the candidate biological cells. Applyingthe candidate treatment regimens may include applying any superpositionof various candidate treatment compounds, various candidate treatmentconcentrations, various candidate treatment durations, or variouscandidate treatment conditions (e.g., temperatures) to various candidatebiological cells. In some embodiments, the various candidate treatmentregimens may be separated by location on the multi-well sample plate.For example, candidate biological cells in a first row on a multi-wellsample plate may be treated with a first candidate treatment compoundand candidate biological cells in a second row on the multi-well sampleplate may be treated with a second candidate treatment compound. Otheralternative treatment arrangements, such as those described with respectto FIG. 9, are also possible.

At block 1106, the method 1100 includes recording a target image of thetarget biological cell. Recording the target image may include using acamera or a CCD to record an image of a well, or wells, of themulti-well sample plate containing target biological cells. Further, insome embodiments, the camera may have one or more associated opticalfilters configured to allow the transmission of only a range ofwavelengths to the camera. In some embodiments, recording the targetimage may include storing the target image within a memory (e.g., anon-volatile memory, such as an external hard drive or a secure digital,SD, card).

At block 1108, the method 1100 includes recording candidate images ofeach of the candidate biological cells, each having a respectivecandidate phenotype arising in response to the candidate treatmentregimen being applied. One or more of the candidate phenotypes may bedifferent from a phenotype exhibited by the respective candidatebiological cell prior to treatment. For example, a candidate biologicalcell may have initially exhibited an unhealthy phenotype, but aftertreatment using a candidate treatment regimen has a candidate phenotypecloser to a healthy phenotype.

Similar to block 1106, recording the images of each of the candidatebiological cells may include using a camera or a CCD to record an imageof a well, or wells, of the multi-well sample plate containing candidatebiological cells. Further, in some embodiments, the camera may have oneor more associated optical filters configured to allow the transmissionof only a range of wavelengths to the camera. Also, recording thecandidate image may include storing the candidate image within a memory.

At block 1110, the method 1100 includes receiving, by the computingdevice, the target image and the candidate images. The target image orthe candidate images may be transmitted directly from a camera to thecomputing device, in some embodiments (e.g., over WiFi, over Bluetooth®,or via a USB cable).

In alternate embodiments, the computing device may receive the targetimage and/or the candidate images through communication with anothercomputing device. For example, the computing device may receive thetarget image and/or the candidate images from a mobile computing device(e.g., a mobile phone equipped with a camera that recorded the targetimage or the candidate images), a tablet computing device, or a personalcomputing device (e.g., a laptop computing device). The images may bereceived by the computing device via transmission using the publicInternet, in some embodiments. The computing device may receive thetarget image or the candidate images via an app or through email, invarious embodiments.

In some embodiments, one or more of the images received by the computingdevice may be accompanied by image metadata. For example, image metadatamay include when the image was recorded, the number of channels in theimage, the bit-depth of each channel in the image, to which wavelengthranges or cellular components each channel in the image corresponds, apredetermined phenotype associated with the cells in the image, atreatment regimen provided to the cells in the target image, themechanisms of action occurring in the cells of the target image, a rowof a multi-well sample plate from which the image was recorded, a columnof a multi-well sample plate from which the image was recorded, or ananatomical region of a patient from which the cells in the image wereacquired.

At block 1112, the method 1100 includes obtaining, by the computingdevice, a semantic embedding associated with the target image. Likewise,at block 1114, the method 1100 includes obtaining, by the computingdevice for each candidate image, a semantic embedding associated withthe respective candidate image. The semantic embeddings associated withthe target image or a respective candidate image may be generated usinga machine-learned, deep metric network model. The machine-learned, deepmetric network model may have been previously trained using consumerphotographic training data. For example, the consumer photographictraining data may include three-image sets (e.g., similar to thethree-image set illustrated in FIG. 1). The three-image sets may bebased upon internet search results and selections (e.g., by internetusers). The internet search results and selections may allow thecomputing device to train the machine-learned, deep metric network modelto identify similarities and differences between images. In someembodiments, particular features of images will be separated intodimensions (e.g., 64 dimensions) of the semantic embedding during thetraining of the machine-learned, deep metric network.

Further, in some embodiments, obtaining the semantic embeddingassociated with the target image or the respective candidate image mayinclude a similar process to the process illustrated in FIG. 7A. Forexample, obtaining the semantic embedding associated with the targetimage may include separating the channels of the target image, obtaininga semantic embedding for each channel of the target image, andconcatenating each channel of the target image into a single semanticembedding. The concatenated semantic embedding may have more dimensionsthan the single-channel semantic embeddings (e.g., 192 dimensions for aconcatenated semantic embedding of a candidate image having threechannels, each channel having a 64 dimension single-channel semanticembedding).

At block 1116, the method 1100 includes determining, by the computingdevice, a similarity score for each candidate image. Determining thesimilarity score for a respective candidate image includes computing, bythe computing device, a vector distance in a multi-dimensional spacedescribed by the semantic embeddings between the respective candidateimage and the target image. The similarity score for each candidateimage represents a degree of similarity between the target phenotype andthe respective candidate phenotype. Similar to method 1000 of FIG. 10,in some embodiments, the method 1100 may include further analysis of thesimilarity scores.

At block 1118, the method 1100 includes selecting, by the computingdevice, a preferred treatment regimen among a variety of candidatetreatment regimens based on the similarity scores. For example, thecandidate image having the similarity score with the largest value (or,in some embodiments, the smallest value) may be selected. The candidateimage having the similarity score with the largest value may correspondto a candidate biological cell that has a candidate phenotype that ismost similar to the target phenotype. If, for example, the targetphenotype is healthy, and the candidate biological cells were unhealthy,the resulting greatest similarity score may correspond to a biologicalcell that received a candidate treatment regimen that made the cellclosest to a healthy phenotype among the candidate biological cells. Inanother example, if the target phenotype was an unhealthy phenotype, thecandidate image having the lowest similarity score may be selected,which would then correspond to a preferred treatment regimen whichyields a candidate phenotype that is most dissimilar from the targetphenotype.

In alternate embodiments, the method 1100 may include additional blocks.In some embodiments, for example, the method 1100 may additionallyinclude administering the preferred treatment regimen to a patient.Administering the preferred treatment regimen to the patient may includeproviding an anatomical region of a patient with a treatment compound ata given concentration and/or for a given duration. The treatmentcompound, concentration, and/or duration may correspond to the treatmentcompound, concentration, and/or duration applied to the candidatebiological cells in block 1104 that ultimately corresponded to thecandidate image with the highest similarity score, as determined inblock 1116.

III. EXAMPLE SYSTEMS

FIG. 12 is an illustration of a system 1200, according to exampleembodiments. The system 1200 may include a camera 1210, a multi-wellsample plate 1220, a computing device 1230 (including a processor 1240and a memory 1250), and a server 1260. The multi-well plate 1220 mayinclude multiple wells 1222. Further, the multi-well sample plate 1220may be analogous to the multi-well sample plate 900 illustrated in FIG.9.

The camera 1210 may include one or more image sensors (e.g., CCDs). Thecamera 1210 may also include a lens 1212 and one or more optical filters1214. The optical filter may only pass light through to the lens 1212within a certain wavelength range. The wavelength range may correspondto one or more targeted regions of one or more biological cells withinone or more wells 1222 in the multi-well sample plate 1220. For example,a nucleus of the biological cells within one of the sample plates may bedyed using a dye of a particular color. The optical filter 1214 may thenpermit only wavelengths corresponding to the dye to pass to the lens1212 of the camera 1210. In this way, only those regions of thebiological cell being targeted may be recorded by the camera 1210,thereby reducing noise or unnecessary image content.

In another example embodiment, certain cellular organelles may betargeted by one or more fluorophores. The fluorophores may emit lightwithin a first specific wavelength range when excited by radiationwithin a second wavelength range. Thus, in such embodiments, the system1200 may additionally include an excitation source (e.g., a laser) thatemits light within the second wavelength range to excite thefluorophores.

In some embodiments, multiple optical filters 1214 may be cascaded toabsorb and/or reflect light of various wavelength ranges. Additionallyor alternately, the optical filter 1214 may be interchangeable. Forexample, as the camera 1210 is scanned over various wells 1222 of themulti-well sample plate 1220, the optical filter 1214 may be removed orswapped for various alternate optical filters (e.g., to analyze varioustargeted regions within various wells 1222 corresponding to variouswavelength ranges).

As illustrated, the camera 1210 is communicatively coupled to thecomputing device 1230. Such a communicative coupling may be implementedusing WiFi, over Bluetooth®, or via wireline interface (e.g., a USBcable), in various embodiments. Alternatively, in some embodiments, thecamera 1210 may be coupled to the computing device 1230 over the publicInternet. For example, the camera 1210 may be a camera attached to orintegrated in a mobile computing device (e.g., a cellular phone). Themobile computing device may access the public Internet to transmitimages (e.g., candidate images or target images of biological cells) tothe computing device 1230. In some embodiments, the camera 1210 mayadditionally or alternatively be communicatively coupled to the server1260. For example, in some embodiments, the camera 1210 may transmitimages to the server 1260, the server 1260 may perform image processing(e.g., a creation of semantic embeddings using a machine-learned, deepmetric network model and a comparison of the semantic embeddings toobtain similarity scores), and the server 1260 may then transmit theresulting similarity scores to the computing device 1230.

The computing device 1230, as illustrated, includes a processor 1240 anda memory 1250. The memory 1250 includes instructions 1252 storedthereon. The memory 1250 may include volatile memory (e.g., RAM) and/ornon-volatile memory (e.g., a hard drive). The memory 1250 may also beinternally communicatively coupled to the processor 1240 (e.g., over asystem bus). The processor 1240 may be configured to execute theinstructions 1252 stored in the memory 1250 (e.g., to perform variouscomputing tasks). Additionally or alternatively, the memory 1250 maystore images (e.g., recorded by the camera 1210) and semantic embeddingsassociated with the images. The memory 1250 may further store amachine-learned, deep metric network model used to generate semanticembeddings from images.

The computing device 1230, as illustrated, may also be communicativelycoupled to the server 1260 (e.g., over the public Internet). In someembodiments, the server 1260 (alternative to or in addition to thememory 1250) may store the machine-learned, deep metric network modelused to generate semantic embeddings. In such embodiments, themachine-learned, deep metric network model may be accessed by thecomputing device 1230 in order for the processor 1240 to generatesemantic embeddings. Further, the server 1260 may store semanticembeddings generated from images (e.g., target images or candidateimages). Such semantic embeddings may be generated by the server 1260,itself, or by the processor 1240 using the machine-learned, deep metricnetwork model. The semantic embeddings may be transmitted from theserver 1260 to the computing device 1230 such that the processor 1240can perform comparisons of the semantic embeddings to obtain similarityscores. The server 1260 may also store similarity scores from previousimage comparisons (e.g., comparisons performed by the processor 1240).

IV. ADDITIONAL PROCESSES

FIG. 13 is an illustration of a method 1300, according to exampleembodiments. In some embodiments, various blocks may represent a module,a segment, or a portion of program code, which includes one or moreinstructions executable by a processor or computing device forimplementing specific logical operations or steps. The program code maybe stored on any type of computer-readable medium, such as a storagedevice included in a disk or hard drive. The computer-readable mediummay include volatile memory, such as a register memory, a processorcache, and/or a RAM. Additionally or alternatively, thecomputer-readable medium may include non-volatile memory, such assecondary or persistent long-term storage, like ROM, optical or magneticdisks, and CD-ROM, for example. In addition, one or more blocks in FIG.13 may represent circuitry that is wired to perform the specific logicaloperations.

In some embodiments, the method 1300 may include additional blocksoccurring before, in between, or after the blocks illustrated in FIG.13. Further, in some embodiments, one or more of the blocks illustratedin FIG. 13 may be repeated one or more times. In alternate embodiments,the order of various blocks within the method may be rearranged withoutdeparting from the scope of the method.

At block 1302, the method 1300 includes receiving, by a computingdevice, a plurality of candidate images of candidate biological cellseach having a respective candidate phenotype.

At block 1304, the method 1300 includes obtaining, by the computingdevice for each of the plurality of candidate images, a semanticembedding associated with the respective candidate image. The semanticembedding associated with the respective candidate image is generatedusing a machine-learned, deep metric network model.

At block 1306, the method 1300 includes grouping, by the computingdevice, the plurality of candidate images into a plurality of phenotypicstrata based on their respective semantic embeddings. Grouping theplurality of candidate images into the plurality of phenotypic stratabased on their respective semantic embeddings includes computing, by thecomputing device for each candidate image, values for one or moredimensions in a multi-dimensional space described by the semanticembeddings. Grouping the plurality of candidate images into theplurality of phenotypic strata based on their respective semanticembeddings also includes comparing, by the computing device, the valuesfor at least one of the one or more dimensions of each candidate image.

In some embodiments, the plurality of phenotypic strata may include afirst phenotypic stratum and a second phenotypic stratum. Further, thefirst phenotypic stratum may correspond to a first stage of a disease(e.g., first stage of Parkinson's disease) and the second phenotypicstratum may correspond to a second stage of the disease (e.g., secondstage of Parkinson's disease). Alternatively, in some embodiments, thefirst phenotypic stratum may correspond to a first type of a disease(e.g., Hemoglobin SS form of sickle cell disease (i.e., sickle cellanemia) or sporadic Parkinson's disease) and the second phenotypicstratum may correspond to a second type of the disease (e.g., HemoglobinSC form of sickle cell disease or familial genetic Parkinson's disease).

Additionally or alternatively, in some embodiments, grouping theplurality of candidate images into the plurality of phenotypic stratabased on their respective semantic embeddings may include determining,based on the values for at least one of the one or more dimensions, athreshold value for the at least one dimension. The threshold value maydelineate between a first phenotypic stratum and a second phenotypicstratum.

FIG. 14 is an illustration of a method 1400, according to exampleembodiments. In some embodiments, various blocks may represent a module,a segment, or a portion of program code, which includes one or moreinstructions executable by a processor or computing device forimplementing specific logical operations or steps. The program code maybe stored on any type of computer-readable medium, such as a storagedevice included in a disk or hard drive. The computer-readable mediummay include volatile memory, such as a register memory, a processorcache, and/or a RAM. Additionally or alternatively, thecomputer-readable medium may include non-volatile memory, such assecondary or persistent long-term storage, like ROM, optical or magneticdisks, and CD-ROM, for example. In addition, one or more blocks in FIG.14 may represent circuitry that is wired to perform the specific logicaloperations.

In some embodiments, the method 1400 may include additional blocksoccurring before, in between, or after the blocks illustrated in FIG.14. Further, in some embodiments, one or more of the blocks illustratedin FIG. 14 may be repeated one or more times. In alternate embodiments,the order of various blocks within the method may be rearranged withoutdeparting from the scope of the method.

At block 1402, the method 1400 includes preparing a first set ofcandidate biological cells.

At block 1404, the method 1400 includes applying a variety of candidategenetic modifications to each of the first set of candidate biologicalcells.

At block 1406, the method 1400 includes recording a first set ofcandidate images of each candidate biological cell in the first set ofcandidate biological cells. Each candidate biological cell in the firstset of candidate biological cells has a respective candidate phenotypearising in response to the candidate genetic modification being applied.

At block 1408, the method 1400 includes preparing a second set ofcandidate biological cells.

At block 1410, the method 1400 includes applying a variety of candidatetreatment compounds to each of the second set of candidate biologicalcells.

At block 1412, the method 1400 includes recording a second set ofcandidate images of each candidate biological cell in the second set ofcandidate biological cells. Each candidate biological cell in the secondset of candidate biological cells has a respective candidate phenotypearising in response to the candidate treatment compound being applied.

At block 1414, the method 1400 includes receiving, by a computingdevice, the first set of candidate images and the second set ofcandidate images.

At block 1416, the method 1400 includes obtaining, by the computingdevice for each candidate image in the first set of candidate images andin the second set of candidate images, a semantic embedding associatedwith the respective candidate image. The semantic embedding associatedwith the respective candidate image is generated using amachine-learned, deep metric network model.

At block 1418, the method 1400 includes identifying, by the computingdevice, one or more candidate treatment compounds that mimic one or moreof the genetic modifications based on vector distances in amulti-dimensional space described by the semantic embeddings betweenrespective candidate images.

FIG. 15 is an illustration of a method 1500, according to exampleembodiments. In some embodiments, various blocks may represent a module,a segment, or a portion of program code, which includes one or moreinstructions executable by a processor or computing device forimplementing specific logical operations or steps. The program code maybe stored on any type of computer-readable medium, such as a storagedevice included in a disk or hard drive. The computer-readable mediummay include volatile memory, such as a register memory, a processorcache, and/or a RAM. Additionally or alternatively, thecomputer-readable medium may include non-volatile memory, such assecondary or persistent long-term storage, like ROM, optical or magneticdisks, and CD-ROM, for example. In addition, one or more blocks in FIG.15 may represent circuitry that is wired to perform the specific logicaloperations.

In some embodiments, the method 1500 may include additional blocksoccurring before, in between, or after the blocks illustrated in FIG.15. Further, in some embodiments, one or more of the blocks illustratedin FIG. 15 may be repeated one or more times. In alternate embodiments,the order of various blocks within the method may be rearranged withoutdeparting from the scope of the method.

At block 1502, the method 1500 includes applying a respectiveperturbation to each of a plurality of subjects in a controlledenvironment.

At block 1504, the method 1500 includes producing a respective visualrepresentation for each of the perturbed subjects using at least oneimaging modality.

At block 1506, the method 1500 includes obtaining, by a computing devicefor each of the respective visual representations, a correspondingsemantic embedding associated with the respective visual representation.The semantic embedding associated with the respective visualrepresentation is generated using a machine-learned, deep metric networkmodel.

At block 1508, the method 1500 includes classifying, by the computingdevice based on the corresponding semantic embedding, each of the visualrepresentations into one or more groups.

In some embodiments of the method 1500, the at least one imagingmodality may include computed tomography, magnetic resonance imaging,positron emission tomography, ultrasound, x-ray computed tomography,x-ray diffraction, fluoroscopy, projectional radiography, single-photonemission computed tomography, scintigraphy, elastography, photoacousticimaging, near-infrared spectroscopy, magnetic particle imaging,optoacoustic imaging, diffuse optical tomography, Raman spectroscopy,fluorescent microscopy, confocal microscopy, two-photon microscopy,hyperspectral analysis, transmission microscopy, electromagneticscanning, differential interference contrast microscopy, multiphotonmicroscopy, dark-field microscopy, quantitative phase-contrastmicroscopy, near-field scanning optical microscopy, photo-activatedlocalization microscopy, second harmonic imaging, holography, scanningelectron microscopy, and/or tunneling electron microscopy.

In some embodiments of the method 1500, the plurality of subjects mayinclude biological cells, fibroblasts, malaria cells, yeast cells, yeastcultures, bacteria, bacterial cultures, fungus, fungal cultures, cancercells, blood cells, malarial parasites, mitochondria, nuclei, axons,dendrites, induced pluripotent stem cells, biological cells from a givenregion of an organism, biological cells from a given tissue of anorganism, biological cells from a given organ of an organism, biologicalcells from a given system of an organism, biological cell ensembles,tissues, organs, organoids, biological systems, organisms, groups oforganisms, ecosystems, chemical compounds, crystals, metallic glasses,mixtures of metallic salts, semiconductors, metals, dielectrics,graphene, microelectromechanical systems (MEMS), and/ornanoelectromechanical systems (NEMS).

In some embodiments of the method 1500, applying the respectiveperturbation to each of the plurality of subjects includes at least oneof: adding one or more small molecule compounds to at least one of theplurality of subjects; applying a candidate treatment compound to atleast one of the plurality of subjects; applying a genetic modificationto at least one of the plurality of subjects; allowing a predeterminedamount of time to elapse for at least one of the plurality of subjects;illuminating at least one of the plurality of subjects with light of apredetermined wavelength; heating or cooling at least one of theplurality of subjects to a predetermined temperature; exposing at leastone of the plurality of subjects to another subject of the plurality ofsubjects; introducing at least one of the plurality of subjects to adifferent environment; or applying a predetermined force to one or moreregions of at least one of the plurality of subjects.

In some embodiments of the method 1500, each of the respective visualrepresentations may include a three-dimensional visual representation.Additionally or alternatively, the machine-learned, deep metric networkmodel may be trained using three-dimensional training data.

In some embodiments of the method 1500, each of the respective visualrepresentations may include a videographic representation. Additionallyor alternatively, the machine-learned, deep metric network model wastrained using videographic training data.

In some embodiments of the method 1500, producing the respective visualrepresentations may include performing a patch selection to identifyeach of the respective perturbed subjects. In some embodiments, thepatch selection may be machine-learned and/or may include an attentionmechanism used to define which regions of a preliminary visualrepresentation are of threshold interest level to be selected by thepatch selection.

In some embodiments of the method 1500, obtaining the correspondingsemantic embedding associated with the respective visual representationsmay include incorporating data captured during one or more respectivesupplementary measurements of a given measurement type. In someembodiments, the data captured during one or more respectivesupplementary measurements may include experimental metadata (e.g.,information about how the experiment was performed, in what spatiallocation the experiment was performed, subject provenance for subjectsof the experiment, or preparation history for the experiment),transcriptomic data, genomic data, proteomic data, metabolomic data,lipidomic data, bulk semiconductor material properties, and/or patientdiagnostic data.

Additionally or alternatively, the machine-learned, deep metric networkmodel may be trained using training data that includes data of the givenmeasurement type. Each visual representation may include a plurality ofchannels. Further, incorporating data captured during one or morerespective supplementary measurements may include: retrieving, by thecomputing device, each channel of the respective visual representation;generating, by the computing device using the machine-learned, deepmetric network model, a respective single-channel semantic embedding foreach channel of the respective visual representation; generating, by thecomputing device using the machine-learned, deep metric network model,an additional single-channel semantic embedding for the data capturedduring one or more respective supplementary measurements; andconcatenating, by the computing device, the single-channel semanticembeddings for each channel of the respective visual representation withthe additional single-channel semantic embedding into a multi-channelsemantic embedding.

In other embodiments, the machine-learned, deep metric network model maybe trained using training data that includes data of the givenmeasurement type. Further, incorporating data captured during one ormore respective supplementary measurements may include: generating afirst preliminary semantic embedding for the respective visualrepresentation using the machine-learned, deep metric network model;generating a second preliminary semantic embedding for the data capturedduring one or more respective supplementary measurements using themachine-learned, deep metric network model; and generating a compositesemantic embedding using an additional machine-learned model. Thecomposite semantic embedding may include a hybrid of the firstpreliminary semantic embedding and the second preliminary semanticembedding.

In still other embodiments, the machine-learned, deep metric networkmodel was trained using training data that includes data of the givenmeasurement type. Further, incorporating data captured during one ormore respective supplementary measurements may include: generating afirst preliminary semantic embedding for the respective visualrepresentation using the machine-learned, deep metric network model;generating a second preliminary semantic embedding for the data capturedduring one or more respective supplementary measurements using themachine-learned, deep metric network model; and generating a compositesemantic embedding by applying a mathematical operation to correspondingdimensions of the first preliminary semantic embedding and the secondpreliminary semantic embedding.

In some embodiments of the method 1500, classifying each of the visualrepresentations into one or more groups includes performing vectorarithmetic on the corresponding semantic embedding to determine to whichof the one or more groups the respective visual representation belongs.

In some embodiments of the method 1500, classifying each of the visualrepresentations into one or more groups includes applying aclassification model to each of the corresponding semantic embeddings.The classification model may be trained to identify responses to therespective perturbations and disregard statistical variations presentacross the plurality of subjects.

Additionally or alternatively, the classification model may be trainedusing a human-in-the-loop procedure to identify which variationscorrespond to notable differences and which variations are attributableto randomness.

In some embodiments of the method 1500, classifying the visualrepresentations into one or more groups may include: identifying one ormore values corresponding to one or more respective dimensions of thesemantic embedding associated with the respective visual representation;and determining, based on the one or more values, to which of anenumerated list of groups the respective visual representation belongs.

In some embodiments of the method 1500, classifying the visualrepresentations into one or more groups may include extractinginformation about the respective visual representation based on atopology within the semantic embedding associated with the respectivevisual representation. The topology may have been enforced duringtraining of the machine-learned, deep metric network model. Further, insome embodiments, the topology is linear, elliptical, circular,polygonal, spherical, ellipsoidal, cylindrical, conical, hyperboloidal,paraboloidal, toroidal, pyramidal, polyhedral, or defined in adimensional space having greater than three dimensions.

In some embodiments of the method 1500, the machine-learned, deep metricnetwork model is trained using a plurality of photographic images astraining data. Further, the photographic images may be query resultsranked based on selections. Additionally, rankings of the photographicimages may be used in the training of the machine-learned, deep metricnetwork model to determine image similarity between two images.

In some embodiments, the method 1500 may also include training themachine-learned, deep metric network model. Training themachine-learned, deep metric network model may include receiving, by thecomputing device, a series of three-image sets as training data. Eachthree-image set may include a query image, a positive image, and anegative image. Further, the query image, the positive image, and thenegative image may be query results ranked in comparison with oneanother based on selections. Additionally, the selections may indicatethat a similarity between the query image and the positive image isgreater than a similarity between the query image and the negativeimage. Still further, training the machine-learned, deep metric networkmodel may include refining, by the computing device, themachine-learned, deep metric network model based on each three-image setto account for image components of the query image, the positive image,and the negative image.

In some embodiments, the method 1500 may include determining, by thecomputing device, a preferred perturbation from among the respectiveperturbations based on the one or more groups into which each of thevisual representations are classified.

FIG. 16 is an illustration of a method 1600, according to exampleembodiments. In some embodiments, various blocks may represent a module,a segment, or a portion of program code, which includes one or moreinstructions executable by a processor or computing device forimplementing specific logical operations or steps. The program code maybe stored on any type of computer-readable medium, such as a storagedevice included in a disk or hard drive. The computer-readable mediummay include volatile memory, such as a register memory, a processorcache, and/or a RAM. Additionally or alternatively, thecomputer-readable medium may include non-volatile memory, such assecondary or persistent long-term storage, like ROM, optical or magneticdisks, and CD-ROM, for example. In addition, one or more blocks in FIG.16 may represent circuitry that is wired to perform the specific logicaloperations.

In some embodiments, the method 1600 may include additional blocksoccurring before, in between, or after the blocks illustrated in FIG.16. Further, in some embodiments, one or more of the blocks illustratedin FIG. 16 may be repeated one or more times. In alternate embodiments,the order of various blocks within the method may be rearranged withoutdeparting from the scope of the method.

At block 1602, the method 1600 includes, for each respective subject ofa plurality of subjects, receiving, by a computing device, acorresponding visual representation of the respective subject that wasproduced using at least one imaging modality after application of arespective perturbation to the respective subject in a controlledenvironment.

At block 1604, the method 1600 includes obtaining, by the computingdevice, a semantic embedding associated with the visual representationof the perturbed subject. The semantic embedding associated with thevisual representation of the perturbed subject is generated using amachine-learned, deep metric network model.

At block 1606, the method 1600 includes determining, by the computingdevice based on the semantic embedding, what type of perturbation wasapplied to the perturbed subject.

V. CONCLUSION

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. While various aspects and embodiments have beendisclosed herein, other aspects and embodiments will be apparent. Thevarious aspects and embodiments disclosed herein are for purposes ofillustration only and are not intended to be limiting, with the truescope being indicated by the following claims.

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The exampleembodiments described herein and in the figures are not meant to belimiting. Other embodiments can be used, and other changes can be made,without departing from the scope of the subject matter presented herein.It will be readily understood that the aspects of the presentdisclosure, as generally described herein, and illustrated in thefigures, can be arranged, substituted, combined, separated, and designedin a wide variety of different configurations, all of which areexplicitly contemplated herein.

The particular arrangements shown in the figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in thefigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed:
 1. A method, comprising: receiving, by a computingdevice, a target image of a target biological cell having a targetphenotype; obtaining, by the computing device, a semantic embeddingassociated with the target image, wherein the semantic embeddingassociated with the target image is generated using a machine-learned,deep metric network model; obtaining, by the computing device for eachof a plurality of candidate images of candidate biological cells eachhaving a respective candidate phenotype, a semantic embedding associatedwith the respective candidate image, wherein the semantic embeddingassociated with the respective candidate image is generated using themachine-learned, deep metric network model; identifying, by thecomputing device for each of the semantic embeddings, commonmorphological variations; reducing, by the computing device for each ofthe semantic embeddings based on the identified common morphologicalvariations, effects of nuisances; and determining, by the computingdevice, a similarity score for each candidate image, wherein thesimilarity score for each candidate image represents a degree ofsimilarity between the target phenotype and the respective candidatephenotype.
 2. The method of claim 1, wherein the machine-learned, deepmetric network model was trained using a plurality of photographicimages as training data, wherein the photographic images are queryresults ranked based on selections, and wherein rankings of thephotographic images are used in the training of the machine-learned,deep metric network model to determine image similarity between twoimages.
 3. The method of claim 1, further comprising training themachine-learned, deep metric network model, wherein training themachine-learned, deep metric network model comprises: receiving, by thecomputing device, a series of three-image sets as training data, whereineach three-image set comprises a query image, a positive image, and anegative image, wherein the query image, the positive image, and thenegative image are query results ranked in comparison with one anotherbased on selections, and wherein the selections indicate that asimilarity between the query image and the positive image is greaterthan a similarity between the query image and the negative image; andrefining, by the computing device, the machine-learned, deep metricnetwork model based on each three-image set to account for imagecomponents of the query image, the positive image, and the negativeimage.
 4. The method of claim 1, further comprising: determining, by thecomputing device, a threshold similarity score; and determining, by thecomputing device, those candidate images having similarity scoresgreater than or equal to the threshold similarity score.
 5. The methodof claim 4, wherein the target phenotype is a healthy phenotype, whereinthe candidate images having similarity scores greater than or equal tothe threshold similarity score have respective candidate phenotypescorresponding to the healthy phenotype, and wherein those candidateimages having similarity scores less than the threshold similarity scorehave respective candidate phenotypes corresponding to an unhealthyphenotype.
 6. The method of claim 4, wherein the target phenotype is anunhealthy phenotype, wherein the candidate images having similarityscores greater than or equal to the threshold similarity score haverespective candidate phenotypes corresponding to the unhealthyphenotype, and wherein those candidate images having similarity scoresless than the threshold similarity score have respective candidatephenotypes corresponding to a healthy phenotype.
 7. The method of claim1, further comprising: determining, by the computing device, a maximumsimilarity score among all of the candidate images, wherein the maximumsimilarity score corresponds to a candidate image of a candidatebiological cell having a candidate phenotype that is most similar to thetarget phenotype.
 8. The method of claim 7, further comprising:determining, by the computing device, a treatment regimen for a patientbased on the candidate biological cell having the candidate phenotypethat is most similar to the target phenotype.
 9. The method of claim 1,wherein identifying the common morphological variations comprisesobtaining, by the computing device for each of a plurality of controlgroup images of control group biological cells having control groupphenotypes, a semantic embedding associated with the respective controlgroup image, wherein the semantic embedding associated with therespective control group image is generated using the machine-learned,deep metric network model, wherein reducing the effects of nuisancescomprises normalizing, by the computing device, the semantic embeddingsassociated with the candidate images, and wherein normalizing comprises:computing, by the computing device, eigenvalues and eigenvectors of acovariance matrix defined by the values of each dimension of thesemantic embeddings associated with the control group images usingprincipal component analysis; scaling, by the computing device, valuesof each dimension of the semantic embeddings associated with the controlgroup images by a respective dimensional scaling factor such that eachdimension is zero-centered and has unit variance; and scaling, by thecomputing device, values of each corresponding dimension of the semanticembeddings associated with the candidate images by the respectivedimensional scaling factor.
 10. The method of claim 9, whereinnormalizing, by the computing device, the semantic embeddings associatedwith the candidate images negates an influence of the commonmorphological variations on the similarity scores.
 11. The method ofclaim 10, wherein the common morphological variations comprisevariations in cellular size, variations in nuclear size, variations incellular shape, variations in nuclear shape, variations in nuclearcolor, variations in nuclear size relative to cellular size, orvariations in nuclear location within a respective cell.
 12. The methodof claim 1, further comprising: obtaining the candidate biologicalcells, wherein the candidate biological cells initially exhibitunhealthy phenotypes; treating the candidate biological cells withvarious concentrations of various candidate treatment compounds;recording the candidate images of the candidate biological cells; andtransmitting the plurality candidate images to the computing device. 13.The method of claim 1, wherein the target biological cell was acquiredfrom a first anatomical region of a patient during a biopsy, and whereinthe candidate biological cells comprise biological cells acquired fromanatomical regions of the patient during the biopsy other than the firstanatomical region.
 14. The method of claim 1, wherein the semanticembedding associated with the target image and the semantic embeddingsassociated with the candidate images comprise 64-dimensional embeddingsfor each channel of a corresponding image.
 15. The method of claim 1,further comprising: scaling, by the computing device, the target imageand each of the candidate images such that an image size of the targetimage and images sizes of each of the candidate images match an imagesize interpretable using the machine-learned, deep metric network model.16. The method of claim 15, wherein scaling the target image and each ofthe candidate images comprises, for each respective image: determining,by the computing device, a location of a cellular nucleus within therespective image; and cropping, by the computing device, the respectiveimage based on a rectangular box centered on the cellular nucleus. 17.The method of claim 1, wherein obtaining the semantic embeddingassociated with the target image or one of the respective candidateimages comprises: retrieving, by the computing device, each channel of arespective image; obtaining, by the computing device, a semanticembedding for each channel of the respective image, wherein the semanticembedding for each channel of the respective image is generated usingthe machine-learned, deep metric network model; and concatenating, bythe computing device, the semantic embeddings for each channel of therespective image into a single semantic embedding.
 18. The method ofclaim 17, wherein each channel in the target image and each of thecandidate images corresponds to a predetermined range of wavelengthsdetectable by one or more cameras, each having one or more associatedoptical filters, wherein each of the one or more cameras is configuredto record the target image or at least one of the plurality of candidateimages, and wherein one or more investigated regions of the targetbiological cell or the candidate biological cells emit light within thepredetermined ranges of wavelengths.
 19. The method of claim 18, whereinthe one or more investigated regions of the target biological cell orthe candidate biological cells emit light within the predeterminedranges of wavelengths due to a fluorescent compound that targets the oneor more investigated regions and is excited by an electromagneticexcitation source, a chemical dye that targets the one or moreinvestigated regions, or a chemiluminescent compound that targets theone or more investigated regions.
 20. The method of claim 18, whereinthe one or more investigated regions correspond to a location of anucleus, a nucleolus, a cytoskeleton, a mitochondrion, a cellularmembrane, an endoplasmic reticulum, a golgi body, a ribosome, a vesicle,a vacuole, a lysosome, or a centrosome within the target biological cellor the candidate biological cells.
 21. The method of claim 1, whereinthe plurality of candidate cells each correspond to a candidatemechanism of action.
 22. The method of claim 1, wherein obtaining, bythe computing device, the semantic embedding associated with the targetimage comprises generating, by an autoencoder, the semantic embeddingassociated with the target image, and wherein obtaining, by thecomputing device for each candidate image, the semantic embeddingassociated with the respective candidate image comprises generating, bythe autoencoder for each candidate image, the semantic embeddingassociated with the respective candidate image.
 23. A non-transitory,computer-readable medium having instructions stored thereon, wherein theinstructions, when executed by a processor, cause the processor toexecute a method, comprising: receiving, by the processor, a targetimage of a target biological cell having a target phenotype; obtaining,by the processor, a semantic embedding associated with the target image,wherein the semantic embedding associated with the target image isgenerated using a machine-learned, deep metric network model; obtaining,by the processor for each of a plurality of candidate images ofcandidate biological cells each having a respective candidate phenotype,a semantic embedding associated with the respective candidate image,wherein the semantic embedding associated with the respective candidateimage is generated using the machine-learned, deep metric network model;identifying, by the processor for each of the semantic embeddings,common morphological variations; reducing, by the processor for each ofthe semantic embeddings based on the identified common morphologicalvariations, effects of nuisances; and determining, by the processor, asimilarity score for each candidate image, wherein the similarity scorefor each candidate image represents a degree of similarity between thetarget phenotype and the respective candidate phenotype.
 24. A method,comprising: preparing a multi-well sample plate with a target biologicalcell having a target phenotype and candidate biological cells; applyinga variety of candidate treatment regimens to each of the candidatebiological cells; recording a target image of the target biologicalcell; recording candidate images of each of the candidate biologicalcells, each having a respective candidate phenotype arising in responseto the candidate treatment regimen being applied; receiving, by acomputing device, the target image and the candidate images; obtaining,by the computing device, a semantic embedding associated with the targetimage, wherein the semantic embedding associated with the target imageis generated using a machine-learned, deep metric network model;obtaining, by the computing device for each candidate image, a semanticembedding associated with the respective candidate image, wherein thesemantic embedding associated with the respective candidate image isgenerated using the machine-learned, deep metric network model;identifying, by the computing device for each of the semanticembeddings, common morphological variations; reducing, by the computingdevice for each of the semantic embeddings based on the identifiedcommon morphological variations, effects of nuisances; determining, bythe computing device, a similarity score for each candidate image,wherein the similarity score for each candidate image represents adegree of similarity between the target phenotype and the respectivecandidate phenotype; and selecting, by the computing device, a preferredtreatment regimen among the variety of candidate treatment regimensbased on the similarity scores.